Add device tree based NUMA support to Arm64
cover Add device tree based NUMA support to Arm64
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern
xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common
tools/libs/util/libxlu_pci.c | 3 +- xen/arch/arm/Kconfig | 10 + xen/arch/arm/Makefile | 2 + xen/arch/arm/arm64/head.S | 9 +- xen/arch/arm/bootfdt.c | 8 +- xen/arch/arm/domain_build.c | 17 +- xen/arch/arm/efi/efi-boot.h | 25 -- xen/arch/arm/numa.c | 162 +++++++++ xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++ xen/arch/arm/platform.c | 4 +- xen/arch/arm/setup.c | 14 + xen/arch/arm/smpboot.c | 37 +- xen/arch/x86/Kconfig | 2 +- xen/arch/x86/numa.c | 421 +---------------------- xen/arch/x86/srat.c | 147 +------- xen/common/Kconfig | 3 + xen/common/Makefile | 1 + xen/common/libfdt/fdt_ro.c | 15 + xen/common/numa.c | 588 ++++++++++++++++++++++++++++++++ xen/common/page_alloc.c | 2 +- xen/drivers/acpi/Kconfig | 3 +- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-arm/numa.h | 33 ++ xen/include/asm-arm/setup.h | 6 + xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/config.h | 1 - xen/include/asm-x86/numa.h | 65 +--- xen/include/asm-x86/setup.h | 1 - xen/include/xen/libfdt/libfdt.h | 25 ++ xen/include/xen/nodemask.h | 2 + xen/include/xen/numa.h | 80 +++++ 31 files changed, 1325 insertions(+), 659 deletions(-) create mode 100644 xen/arch/arm/numa.c create mode 100644 xen/arch/arm/numa_device_tree.c create mode 100644 xen/common/numa.c
Signature · 1 line
2.25.1
RFC PATCH 2/40 xen/arm: Print a 64-bit number in hex from early uart
Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -
In this patch, we increased the print loops and shift bits to make putn print 64-bit number.
--- xen/arch/arm/arm64/head.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +5 -4
@@ -862,17 +862,18 @@ puts: ret ENDPROC(puts) -/* Print a 32-bit number in hex. Specific to the PL011 UART. +/* Print a 64-bit number in hex. Specific to the PL011 UART. * x0: Number to print. * x23: Early UART base address * Clobbers x0-x3 */ +#define PRINT_MASK 0xf000000000000000 putn: adr x1, hex - mov x3, #8 + mov x3, #16 1: early_uart_ready x23, 2 - and x2, x0, #0xf0000000 /* Mask off the top nybble */ - lsr x2, x2, #28 + and x2, x0, #PRINT_MASK /* Mask off the top nybble */ + lsr x2, x2, #60 ldrb w2, [x1, x2] /* Convert to a char */ early_uart_transmit x23, w2 lsl x0, x0, #4 /* Roll it through one nybble at a time */
Signature · 1 line
2.25.1
RFC PATCH 1/40 tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
In reply to Wei Chen| libxlu_pci.c: In function 'xlu_pci_parse_bdf': | libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 32 | pcidev->func = func; | | ~~~~~~~~~~~~~^~~~~~ | libxlu_pci.c:51:29: note: 'func' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~~ | libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 31 | pcidev->dev = dev; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:24: note: 'dev' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 30 | pcidev->bus = bus; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:19: note: 'bus' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 78 | if ( dom & ~0xff ) | | ~~~~^~~~~~~
--- tools/libs/util/libxlu_pci.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
tools/libs/util/libxlu_pci.c +1 -2
@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) {const char *ptr = str;unsigned int colons =0 ; -unsigned int domain, bus, dev, func; +unsigned int domain =0 , bus =0 , dev =0 , func =0 ;int n;/* Count occurrences of ':' to detrmine presence/absence of the 'domain' */
@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) ptr = str;switch (colons) {case 1 : - domain =0 ;if (sscanf (ptr,"%x:%x.%n" , &bus, &dev, &n) !=2 )return ERROR_INVAL ;break ;
Signature · 1 line
2.25.1
RFC PATCH 3/40 xen/x86: Initialize memnodemapsize while faking NUMA node
In reply to Wei ChenWhen system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of _memnodemap. Xen hadn't done it, and Xen should assert in phys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.
In this patch, we set memnodemapsize to ARRAY_SIZE(_memnodemap) to fix it.
--- xen/arch/x86/numa.c | 2 ++ 1 file changed, 2 insertions(+)
xen/arch/x86/numa.c +2 -0
@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap); +nodes_clear (node_online_map);node_set_online (0 );for ( i =0 ; i < nr_cpu_ids; i++ )
Signature · 1 line
2.25.1
RFC PATCH 6/40 xen: decouple NUMA from ACPI in Kconfig
In reply to Wei ChenIn current Xen code only implments x86 ACPI-based NUMA support. So in Xen Kconfig system, NUMA equals to ACPI_NUMA. x86 selects NUMA by default, and CONFIG_ACPI_NUMA is hardcode in config.h.
In this patch series, we introduced device tree based NUMA for Arm. That means we will have two NUMA implemetations, so in this patch we decouple NUMA from ACPI based NUMA in Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.
--- xen/arch/x86/Kconfig | 2 +- xen/common/Kconfig | 3 +++ xen/drivers/acpi/Kconfig | 3 ++- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-x86/config.h | 1 - 5 files changed, 7 insertions(+), 4 deletions(-)
xen/arch/x86/Kconfig +1 -1
@@ -24,7 +24,7 @@ config X86 select HAS_UBSAN select HAS_VPCI if HVM select NEEDS_LIBELF - select NUMA + select ACPI_NUMA config ARCH_DEFCONFIG string
xen/common/Kconfig +3 -0
@@ -67,6 +67,9 @@ config MEM_ACCESS config NEEDS_LIBELF bool +config NUMA + bool + menu "Speculative hardening" config SPECULATIVE_HARDEN_ARRAY
xen/drivers/acpi/Kconfig +2 -1
@@ -5,5 +5,6 @@ config ACPI config ACPI_LEGACY_TABLES_LOOKUP bool -config NUMA +config ACPI_NUMA bool + select NUMA
xen/drivers/acpi/Makefile +1 -1
@@ -3,7 +3,7 @@ obj-y += utilities/ obj-$(CONFIG_X86) += apei/ obj-bin-y += tables.init.o -obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_ACPI_NUMA) += numa.o obj-y += osl.o obj-$(CONFIG_HAS_CPUFREQ) += pmstat.o
xen/include/asm-x86/config.h +0 -1
@@ -31,7 +31,6 @@/* Intel P4 currently has largest cache line (L2 line size is 128 bytes). */ #define CONFIG_X86_L1_CACHE_SHIFT 7 -#define CONFIG_ACPI_NUMA 1#define CONFIG_ACPI_SRAT 1#define CONFIG_ACPI_CSTATE 1
Signature · 1 line
2.25.1
RFC PATCH 5/40 xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
In reply to Wei ChenFrom: Hongda Deng <[email protected]>
In previous patch, we make arch_get_dma_bitsize return 0 when dma_bitsize and platform->dma_bitsize are not set. But this will affect lowmem_bitsize in allocate_memory_11 for domain0. Because this function depends lowmem_bitsize to allocate memory below 4GB.
In current code, when arch_get_dma_bitsize return 0, lowmem_bitsize will be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.
This behavior is inconsistent with what allocate_memory_11 done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before set lowmem_bitsize. Avoid setting lowmem_bitsize to 0 by mistake.
--- xen/arch/arm/domain_build.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
xen/arch/arm/domain_build.c +10 -1
@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain *d,int i;bool lowmem = true; -unsigned int lowmem_bitsize =min (32U ,arch_get_dma_bitsize ()); +unsigned int lowmem_bitsize =arch_get_dma_bitsize ();unsigned int bits; +/* +When dma_bitsize and platform->dma_bitsize are not set, +arch_get_dma_bitsize will return 0. That means this system +doesn't need to reserve memory for DMA. But in order to +meet above requirements, we still need to try to allocate +memory below 4GB for Dom0. +*/ + lowmem_bitsize = lowmem_bitsize ?min (32U , lowmem_bitsize) :32U ; +/* * TODO: Implement memory bank allocation when DOM0 is not direct * mapped
Signature · 1 line
2.25.1
RFC PATCH 4/40 xen/arm: return default DMA bit width when platform is not set
In reply to Wei ChenFrom: Hongda Deng <[email protected]>
In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm, we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function. But in NUMA system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware. Only platform setting can override dma_bitsize. So in this patch, we return default dma_bitsize, when platform and platorm->dma_bitsize are not set.
--- xen/arch/arm/platform.c | 4 +++- xen/common/page_alloc.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)
xen/arch/arm/platform.c +3 -1
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;static bool __init platform_is_compatible (const struct platform_desc *plat) {
@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)unsigned int arch_get_dma_bitsize (void ) { -return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize :32 ; +return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize + : dma_bitsize; }/*
xen/common/page_alloc.c +1 -1
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;integer_param ("dma_bits" , dma_bitsize);/* Offlined page list, protected by heap_lock. */
Signature · 1 line
2.25.1
RFC PATCH 10/40 xen/x86: Move NR_NODE_MEMBLKS macro to common
In reply to Wei ChenNot only x86 ACPI need this macro. Device tree based NUMA also needs this macro to present max memory block number. So we move it from x86 ACPI header file to common NUMA header file.
--- xen/include/asm-x86/acpi.h | 1 - xen/include/xen/numa.h | 1 + 2 files changed, 1 insertion(+), 1 deletion(-)
xen/include/asm-x86/acpi.h +0 -1
@@ -103,7 +103,6 @@ extern unsigned long acpi_wakeup_address;extern s8 acpi_numa;extern int acpi_scan_nodes (u64 start,u64 end); -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)
xen/include/xen/numa.h +1 -0
@@ -11,6 +11,7 @@#define NUMA_NO_DISTANCE 0xFF#define MAX_NUMNODES (1 << NODES_SHIFT) +#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)#define vcpu_to_node (v) (cpu_to_node((v)->processor))
Signature · 1 line
2.25.1
RFC PATCH 12/40 xen/x86: Move numa_initmem_init to common
In reply to Wei ChenThis function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:setup_node_bootmem, numa_init_array and numa_emulation.
As numa_initmem_init has been moved to common, _memnodemap is not used cross files. We can restore _memnodemap to static.
--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)
xen/arch/x86/numa.c +0 -118
@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )
@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn; - -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1 cpu->node -mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ -if (hweight64 (sz) >1 ) - { -u64 x =1 ; -while ( (x <<1 ) < sz ) - x <<=1 ; -if ( x < sz/2 ) -printk (KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n" ); - sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); -for ( i =0 ; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; -if ( i == numa_fake -1 ) - sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; -printk (KERN_INFO "Faking node %d at %" PRIx64"-%" PRIx64" (%" PRIu64"MB)\n" , - i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >>20 ); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); -if ( memnode_shift <0 ) - { - memnode_shift =0 ; -printk (KERN_ERR "No NUMA hash function found. Emulation disabled.\n" ); -return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) -{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ; -#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - -printk (KERN_INFO "%s\n" , - numa_off ?"NUMA turned off" :"No NUMA configuration found" ); - -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;
xen/common/numa.c +121 -1
142 lines not on this page · raw message
xen/include/asm-x86/numa.h +0 -5
@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_init_array (void ); -extern bool numa_off; - -extern int srat_disabled (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );
xen/include/asm-x86/setup.h +0 -1
@@ -17,7 +17,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );
xen/include/xen/numa.h +7 -1
@@ -26,7 +26,6 @@extern int memnode_shift;extern unsigned long memnodemapsize;extern u8 *memnodemap; -extern typeof(*memnodemap) _memnodemap[64];struct node_data {unsigned long node_start_pfn;
@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern void numa_init_array (void ); +extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern bool numa_off; +extern int numa_fake; + +extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
RFC PATCH 7/40 xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Wei ChenOnly Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32. Even in Arm64, users still can disable the CONFIG_NUMA through Kconfig option. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.
--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +4 -0
@@ -5,6 +5,8 @@typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA ) +/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables:
Signature · 1 line
2.25.1
RFC PATCH 8/40 xen/x86: Move NUMA memory node map functions to common
In reply to Wei ChenIn the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from arch/x86 to common.
--- xen/arch/x86/numa.c | 114 -------------------------------- xen/common/Makefile | 1 + xen/common/numa.c | 131 +++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 29 -------- xen/include/xen/numa.h | 35 ++++++++++ 5 files changed, 167 insertions(+), 143 deletions(-) create mode 100644 xen/common/numa.c
xen/arch/x86/numa.c +0 -114
@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -struct node_data node_data[MAX_NUMNODES ]; - -/* Mapping from pdx to node id */ -int memnode_shift; -static typeof(*memnodemap) _memnodemap[64]; -unsigned long memnodemapsize; -u8 *memnodemap; -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { [0 ...NR_CPUS -1 ] =NUMA_NO_NODE };
@@ -58,112 +50,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* -* Given a shift value, try to populate memnodemap[] -* Returns : -* 1 if OK -* 0 if memnodmap[] too small (of shift too small) -* -1 if node overlap or lost ram (shift too big) -*/ -static int __init populate_memnodemap (const struct node *nodes, -int numnodes,int shift,nodeid_t *nodeids) -{ -unsigned long spdx, epdx; -int i, res =-1 ; - -memset (memnodemap,NUMA_NO_NODE , memnodemapsize *sizeof (*memnodemap)); -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; -if ( (epdx >> shift) >= memnodemapsize ) -return 0 ; -do { -if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) -return -1 ; - -if ( !nodeids ) - memnodemap[spdx >> shift] = i; -else - memnodemap[spdx >> shift] = nodeids[i]; - - spdx += (1UL << shift); - }while ( spdx < epdx ); - res =1 ; - } - -return res; -} - -static int __init allocate_cachealigned_memnodemap (void ) -{ -unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); -unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); - - memnodemap =mfn_to_virt (mfn); - mfn <<=PAGE_SHIFT ; - size <<=PAGE_SHIFT ; -printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , - mfn, mfn + size); - memnodemapsize = size /sizeof (*memnodemap); - -return 0 ; -} - -/* -* The LSB of all start and end addresses in the node map is the value of the -* maximum possible shift. -*/ -static int __init extract_lsb_from_nodes (const struct node *nodes, -int numnodes) -{ -int i, nodes_used =0 ; -unsigned long spdx, epdx; -unsigned long bitfield =0 , memtop =0 ; - -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; - bitfield |= spdx; - nodes_used++; -if ( epdx > memtop ) - memtop = epdx; - } -if ( nodes_used <=1 ) - i =BITS_PER_LONG -1 ; -else - i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); - memnodemapsize = (memtop >> i) +1 ; -return i; -} - -int __init compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids) -{ -int shift; - - shift =extract_lsb_from_nodes (nodes, numnodes); -if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) - memnodemap = _memnodemap; -else if (allocate_cachealigned_memnodemap () ) -return -1 ; -printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); - -if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) - { -printk (KERN_INFO "Your memory is not aligned you need to " -"rebuild your hypervisor with a bigger NODEMAPSIZE " -"shift=%d\n" , shift); -return -1 ; - } - -return shift; -}/* initialize NODE_DATA given nodeid and start/end */ void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) {
xen/common/Makefile +1 -0
@@ -54,6 +54,7 @@ obj-y += wait.o obj-bin-y += warning.init.o obj-$(CONFIG_XENOPROF) += xenoprof.o obj-y += xmalloc_tlsf.o +obj-$(CONFIG_NUMA) += numa.o obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma lzo unlzo unlz4 unzstd earlycpio,$(n).init.o)
xen/common/numa.c new file +131 -0
131 lines not on this page · raw message
xen/include/asm-x86/numa.h +0 -29
@@ -26,7 +26,6 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -#define VIRTUAL_BUG_ON (x)extern void numa_add_cpu (int cpu);extern void numa_init_array (void );
@@ -47,34 +46,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -/* Simple perfect hash to map pdx to node numbers */ -extern int memnode_shift; -extern unsigned long memnodemapsize; -extern u8 *memnodemap; - -struct node_data { -unsigned long node_start_pfn; -unsigned long node_spanned_pages; -}; - -extern struct node_data node_data[]; - -static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) -{ -nodeid_t nid; -VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); - nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; -VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); -return nid; -} - -#define NODE_DATA (nid) (&(node_data[nid])) - -#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) -#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) -#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ - NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node);void srat_parse_regions (u64 addr);
xen/include/xen/numa.h +35 -0
@@ -18,4 +18,39 @@ (((d)->vcpu !=NULL && (d)->vcpu[0 ] !=NULL ) \ ?vcpu_to_node ((d)->vcpu[0 ]) :NUMA_NO_NODE ) +/* The following content can be used when NUMA feature is enabled */ +#if defined(CONFIG_NUMA ) + +/* Simple perfect hash to map pdx to node numbers */ +extern int memnode_shift; +extern unsigned long memnodemapsize; +extern u8 *memnodemap; +extern typeof(*memnodemap) _memnodemap[64]; + +struct node_data { +unsigned long node_start_pfn; +unsigned long node_spanned_pages; +}; + +extern struct node_data node_data[]; +#define VIRTUAL_BUG_ON (x) + +static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) +{ +nodeid_t nid; +VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); + nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; +VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); +return nid; +} + +#define NODE_DATA (nid) (&(node_data[nid])) + +#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) +#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) +#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ + NODE_DATA(nid)->node_spanned_pages) + +#endif /* CONFIG_NUMA */ +#endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
RFC PATCH 11/40 xen/x86: Move NUMA nodes and memory block ranges to common
In reply to Wei ChenThese data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.
--- xen/arch/x86/srat.c | 50 ------------------------------------- xen/common/numa.c | 51 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 8 ------ xen/include/xen/numa.h | 15 +++++++++++ 4 files changed, 66 insertions(+), 58 deletions(-)
xen/arch/x86/srat.c +0 -50
@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;static nodemask_t memory_nodes_parsed__initdata ;static nodemask_t processor_nodes_parsed__initdata ; -static struct node nodes[MAX_NUMNODES ]__initdata ;struct pxm2node {unsigned pxm;
@@ -37,9 +36,6 @@ static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =static unsigned node_to_pxm (nodeid_t n); -static int num_node_memblks; -static struct node node_memblk_range[NR_NODE_MEMBLKS ]; -static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);static inline bool node_found (unsigned idx,unsigned pxm)
@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; - -if (nd->start <= start && nd->end >= end && - memblk_nodeid[i] == node) -return 1 ; - } - -return 0 ; -} - -static __init int conflicting_memblks (u64 start,u64 end) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; -if (nd->start == nd->end) -continue ; -if (nd->end > start && nd->start < end) -return i; -if (nd->end == end && nd->start == start) -return i; - } -return -1 ; -} - -static __init void cutoff_node (int i,u64 start,u64 end) -{ -struct node *nd = &nodes[i]; -if (nd->start < start) { - nd->start = start; -if (nd->end < nd->start) - nd->start = nd->end; - } -if (nd->end > end) { - nd->end = end; -if (nd->start > nd->end) - nd->start = nd->end; - } -} -static __init void bad_srat (void ) {int i;
xen/common/numa.c +51 -0
63 lines not on this page · raw message
xen/include/asm-x86/numa.h +0 -8
@@ -17,12 +17,6 @@ extern cpumask_t node_to_cpumask[];#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node]))#define node_to_cpumask (node) (node_to_cpumask[node]) -struct node { -u64 start,end; -}; - -extern int compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids);extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );
xen/include/xen/numa.h +15 -0
@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)extern void numa_add_cpu (int cpu); +struct node { +u64 start,end; +}; + +extern struct node nodes[MAX_NUMNODES ]; +extern int num_node_memblks; +extern struct node node_memblk_range[NR_NODE_MEMBLKS ]; +extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; + +extern int compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids); +extern int conflicting_memblks (u64 start,u64 end); +extern void cutoff_node (int i,u64 start,u64 end); +extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
RFC PATCH 14/40 xen/arm: set NUMA nodes max number to 64 by default
In reply to Wei ChenToday's Arm64 systems can reach or exceed 16 NUMA nodes, so we set the number to 64 to match with x86.
--- xen/include/asm-arm/numa.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +9 -1
@@ -5,7 +5,15 @@typedef u8 nodeid_t ; -#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/ +#define NODES_SHIFT 6 + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0
Signature · 1 line
2.25.1
RFC PATCH 17/40 xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Wei ChenWe need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:DEVICE_TREE_NUMA in this patch for Arm64.
--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)
xen/arch/arm/Kconfig +10 -0
@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xen is an alternative to device tree on ARM64. +config DEVICE_TREE_NUMA + bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" if UNSUPPORTED + depends on ARM_64 + select NUMA + ---help--- + + Non-Uniform Memory Access (NUMA) is a computer memory design used in + multiprocessing, where the memory access time depends on the memory + location relative to the processor. + config GICV3 bool "GICv3 driver" depends on ARM_64 && !NEW_VGIC
Signature · 1 line
2.25.1
RFC PATCH 35/40 xen: enable numa_scan_nodes for device tree based NUMA
In reply to Wei ChenNow, we can use the same function for ACPI and device tree based NUMA to scan memory nodes.
--- xen/common/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/numa.c +1 -1
@@ -381,7 +381,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)return ;#endif -#ifdef CONFIG_ACPI_NUMA +#if defined(CONFIG_ACPI_NUMA ) || defined(CONFIG_DEVICE_TREE_NUMA )if ( !numa_off && !numa_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ) )return ;
Signature · 1 line
2.25.1
RFC PATCH 39/40 xen/x86: move numa_setup to common to support NUMA switch in command line
In reply to Wei ChenXen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.
--- xen/arch/x86/numa.c | 34 ---------------------------------- xen/common/numa.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/xen/numa.h | 1 - 3 files changed, 34 insertions(+), 36 deletions(-)
xen/arch/x86/numa.c +0 -34
@@ -11,7 +11,6 @@#include <xen/nodemask.h> #include <xen/numa.h> #include <xen/keyhandler.h> -#include <xen/param.h> #include <xen/time.h> #include <xen/smp.h> #include <xen/pfn.h>
@@ -19,9 +18,6 @@#include <xen/sched.h> #include <xen/softirq.h> -static int numa_setup (const char *s); -custom_param ("numa" , numa_setup); -#ifndef Dprintk #define Dprintk(x...)#endif
@@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node) cpu_to_node[cpu] = node; } -/* [numa=off] */ -static __init int numa_setup (const char *opt) -{ -if ( !strncmp (opt,"off" ,3 ) ) - numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) - numa_off = false; -#ifdef CONFIG_NUMA_EMU - else if ( !strncmp(opt, "fake=", 5) ) - { - numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); -if ( numa_fake >=MAX_NUMNODES ) - numa_fake =MAX_NUMNODES ; - } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - acpi_numa =-1 ; - } -#endif - else - return -EINVAL; - -return 0 ; -} -/* * Setup early cpu_to_node. *
@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)return 0 ; }__initcall (register_numa_trigger); -
xen/common/numa.c +34 -1
@@ -14,8 +14,12 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> +#include <xen/param.h> #include <asm/acpi.h> +static int numa_setup (const char *s); +custom_param ("numa" , numa_setup); +struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */
@@ -324,7 +328,7 @@ int __init numa_scan_nodes(u64 start, u64 end) }#ifdef CONFIG_NUMA_EMU -int numa_fake__initdata =0 ; +static int numa_fake__initdata =0 ;/* Numa emulation */ static int __init numa_emulation (u64 start_pfn,u64 end_pfn)
@@ -409,3 +413,32 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* architecture specified fallback operations */ arch_numa_init_failed_fallback (); } + +/* [numa=off] */ +static __init int numa_setup (const char *opt) +{ +if ( !strncmp (opt,"off" ,3 ) ) + numa_off = true; +else if ( !strncmp (opt,"on" ,2 ) ) + numa_off = false; +#ifdef CONFIG_NUMA_EMU + else if ( !strncmp(opt, "fake=", 5) ) + { + numa_off = false; + numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); +if ( numa_fake >=MAX_NUMNODES ) + numa_fake =MAX_NUMNODES ; + } +#endif +#ifdef CONFIG_ACPI_NUMA + else if ( !strncmp(opt,"noacpi",6) ) + { + numa_off = false; + acpi_numa =-1 ; + } +#endif + else + return -EINVAL; + +return 0 ; +}
xen/include/xen/numa.h +0 -1
@@ -83,7 +83,6 @@ extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);extern void numa_set_node (int cpu,nodeid_t node);extern int numa_scan_nodes (u64 start,u64 end);extern bool numa_off; -extern int numa_fake;extern s8 acpi_numa;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);
Signature · 1 line
2.25.1
RFC PATCH 20/40 xen/arm: implement node distance helpers for Arm64
In reply to Wei ChenIn current Xen code, __node_distance is a fake API, it always returns NUMA_REMOTE_DISTANCE(20). Now we use a matrix to record the distance between any two nodes. Accordingly, we provide a set_node_distance API to set the distance for any two nodes in this patch.
--- xen/arch/arm/numa.c | 44 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 12 ++++++++++- xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 +- 4 files changed, 56 insertions(+), 3 deletions(-)
xen/arch/arm/numa.c +44 -0
@@ -23,6 +23,11 @@#include <xen/pfn.h> #include <asm/setup.h> +static uint8_t __read_mostly +node_distance_map[MAX_NUMNODES ][MAX_NUMNODES ] = { + {NUMA_REMOTE_DISTANCE } +}; +void numa_set_node (int cpu,nodeid_t nid) { if ( nid >= MAX_NUMNODES ||
@@ -32,6 +37,45 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; } +void __init numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance) +{ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) + { +printk (KERN_WARNING +"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n" , + from, to, distance); +return ; + } + +/* NUMA defines 0xff as an unreachable node and 0-9 are undefined */ +if ( distance >=NUMA_NO_DISTANCE || + (distance >=NUMA_DISTANCE_UDF_MIN && + distance <=NUMA_DISTANCE_UDF_MAX ) || + (from == to && distance !=NUMA_LOCAL_DISTANCE ) ) + { +printk (KERN_WARNING +"Invalid NUMA node distance, from:%d to:%d distance=%d\n" , + from, to, distance); +return ; + } + + node_distance_map[from][to] = distance; +} + +uint8_t __node_distance (nodeid_t from,nodeid_t to) +{ +/* +* Check whether the nodes are in the matrix range. +* When any node is out of range, except from and to nodes are the +* same, we treat them as unreachable (return 0xFF) +*/ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) +return from == to ?NUMA_LOCAL_DISTANCE :NUMA_NO_DISTANCE ; + +return node_distance_map[from][to]; +} +EXPORT_SYMBOL (__node_distance); +void __init numa_init (bool acpi_off) {uint32_t idx;
xen/include/asm-arm/numa.h +11 -1
@@ -12,8 +12,19 @@ typedef u8 nodeid_t;* set the number of NUMA memory block number to 128. */ #define NODES_SHIFT 6 +/* +* In ACPI spec, 0-9 are the reserved values for node distance, +* 10 indicates local node distance, 20 indicates remote node +* distance. Set node distance map in device tree will follow +* the ACPI's definition. +*/ +#define NUMA_DISTANCE_UDF_MIN 0 +#define NUMA_DISTANCE_UDF_MAX 9 +#define NUMA_LOCAL_DISTANCE 10 +#define NUMA_REMOTE_DISTANCE 20extern void numa_init (bool acpi_off); +extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);/* * Temporary for fake NUMA node, when CPU, memory and distance
@@ -21,7 +32,6 @@ extern void numa_init(bool acpi_off);* symbols will be removed. */ extern mfn_t first_valid_mfn; -#define __node_distance (a, b) (20)#else
xen/include/asm-x86/numa.h +0 -1
@@ -21,7 +21,6 @@ extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );void srat_parse_regions (u64 addr); -extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );#endif
xen/include/xen/numa.h +1 -1
@@ -58,7 +58,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) - +extern u8 __node_distance (nodeid_t a,nodeid_t b);extern void numa_add_cpu (int cpu);struct node {
Signature · 1 line
2.25.1
RFC PATCH 37/40 xen: introduce an arch helper to do NUMA init failed fallback
In reply to Wei ChenWhen Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.
--- xen/arch/arm/numa.c | 13 +++++++++++++ xen/common/numa.c | 3 +++ xen/include/asm-arm/numa.h | 1 + xen/include/asm-x86/numa.h | 6 ++++++ 4 files changed, 23 insertions(+)
xen/arch/arm/numa.c +13 -0
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int bank,return 0 ; } + +void __init arch_numa_init_failed_fallback (void ) +{ +int i, j; + +/* Reset all node distance to remote_distance */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) { +for ( j =0 ; j <MAX_NUMNODES ; j++ ) { +numa_set_distance (i, j, + (i == j) ?NUMA_LOCAL_DISTANCE :NUMA_REMOTE_DISTANCE ); + } + } +}
xen/common/numa.c +3 -0
@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 ));setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ); + +/* architecture specified fallback operations */ +arch_numa_init_failed_fallback (); }
xen/include/asm-arm/numa.h +1 -0
@@ -28,6 +28,7 @@ extern s8 device_tree_numa;extern void numa_init (bool acpi_off);extern int numa_device_tree_init (const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance); +extern void arch_numa_init_failed_fallback (void );/* * Temporary for fake NUMA node, when CPU, memory and distance
xen/include/asm-x86/numa.h +6 -0
@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);void srat_parse_regions (u64 addr);unsigned int arch_get_dma_bitsize (void ); +/* Dummy function for numa init failed in numa_initmem_init */ +static inline void arch_numa_init_failed_fallback (void ) +{ +return ; +} +#endif
Signature · 1 line
2.25.1
RFC PATCH 34/40 xen: move numa_scan_nodes from x86 to common
In reply to Wei ChenAfter the previous patches preparations, numa_scan_nodes can be used by Arm and x86. So we move this function from x86 to common. As node_cover_memory will not be used cross files, we restore its static attribute in this patch.
--- xen/arch/x86/srat.c | 52 ------------------------------------ xen/common/numa.c | 54 +++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/acpi.h | 3 --- xen/include/xen/numa.h | 3 ++- 4 files changed, 55 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +0 -52
@@ -361,58 +361,6 @@ void __init srat_parse_regions(u64 addr)pfn_pdx_hole_setup (mask >>PAGE_SHIFT ); } -/* Use the information discovered above to actually set up the nodes. */ -int __init numa_scan_nodes (u64 start,u64 end) -{ -int i; -nodemask_t all_nodes_parsed; - -/* First clean up the node list */ -for (i =0 ; i <MAX_NUMNODES ; i++) -cutoff_node (i, start, end); - -#ifdef CONFIG_ACPI_NUMA -if (acpi_numa <=0 ) -return -1 ; -#endif - -if (!nodes_cover_memory ()) { -bad_srat (); -return -1 ; - } - - memnode_shift =compute_hash_shift (node_memblk_range, num_node_memblks, - memblk_nodeid); - -if (memnode_shift <0 ) { -printk (KERN_ERR -"SRAT: No NUMA node hash function found. Contact maintainer\n" ); -bad_srat (); -return -1 ; - } - -nodes_or (all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed); - -/* Finally register nodes */ -for_each_node_mask (i, all_nodes_parsed) - { -u64 size = nodes[i].end - nodes[i].start; -if ( size ==0 ) -printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); - -setup_node_bootmem (i, nodes[i].start, nodes[i].end); - } -for (i =0 ; i < nr_cpu_ids; i++) { -if (cpu_to_node[i] ==NUMA_NO_NODE ) -continue ; -if (!nodemask_test (cpu_to_node[i], &processor_nodes_parsed)) -numa_set_node (i,NUMA_NO_NODE ); - } -numa_init_array (); -return 0 ; -} -static unsigned node_to_pxm (nodeid_t n) {unsigned i;
xen/common/numa.c +53 -1
66 lines not on this page · raw message
xen/include/asm-x86/acpi.h +0 -3
@@ -101,9 +101,6 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1 -extern s8 acpi_numa; -extern int numa_scan_nodes (u64 start,u64 end); -extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)struct xenpf_enter_acpi_sleep ;
xen/include/xen/numa.h +2 -1
@@ -81,8 +81,10 @@ extern void bad_srat(void);extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);extern void numa_set_node (int cpu,nodeid_t node); +extern int numa_scan_nodes (u64 start,u64 end);extern bool numa_off;extern int numa_fake; +extern s8 acpi_numa;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);
@@ -94,7 +96,6 @@ static inline void clear_node_cpumask(int cpu)extern uint32_t arch_meminfo_get_nr_bank (void );extern int arch_meminfo_get_ram_bank_range (int bank,unsigned long long *start,unsigned long long *end); -extern int nodes_cover_memory (void );#endif /* CONFIG_NUMA */
Signature · 1 line
2.25.1
RFC PATCH 25/40 xen/arm: unified entry to parse all NUMA data from device tree
In reply to Wei ChenIn this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though early_scan_node will invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.
--- xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++--- xen/include/asm-arm/numa.h | 1 + 2 files changed, 29 insertions(+), 3 deletions(-)
xen/arch/arm/numa_device_tree.c +28 -3
@@ -131,7 +131,8 @@ save_memblk: }/* Parse CPU NUMA node info */ -int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +static int __init +device_tree_parse_numa_cpu_node (const void *fdt,int node) {uint32_t nid;
@@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) }/* Parse memory node NUMA info */ -int __init +static int __init device_tree_parse_numa_memory_node (const void *fdt,int node,const char *name,uint32_t addr_cells,uint32_t size_cells) {
@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node, }/* Parse NUMA distance map v1 */ -int __init +static int __init device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) {const struct fdt_property *prop;
@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)return 0 ; } + +static int __init fdt_scan_numa_nodes (const void *fdt, +int node,const char *uname,int depth, +u32 address_cells,u32 size_cells,void *data) +{ +int ret =0 ; + +if (fdt_node_check_type (fdt, node,"cpu" ) ==0 ) + ret =device_tree_parse_numa_cpu_node (fdt, node); +else if (fdt_node_check_type (fdt, node,"memory" ) ==0 ) + ret =device_tree_parse_numa_memory_node (fdt, node, uname, + address_cells, size_cells); +else if (fdt_node_check_compatible (fdt, node, +"numa-distance-map-v1" ) ==0 ) + ret =device_tree_parse_numa_distance_map_v1 (fdt, node); + +return ret; +} + +/* Initialize NUMA from device tree */ +int __init numa_device_tree_init (const void *fdt) +{ +return device_tree_for_each_node (fdt,0 , fdt_scan_numa_nodes,NULL ); +}
xen/include/asm-arm/numa.h +1 -0
@@ -26,6 +26,7 @@ typedef u8 nodeid_t;extern s8 device_tree_numa;extern void numa_init (bool acpi_off); +extern int numa_device_tree_init (const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);/*
Signature · 1 line
2.25.1
RFC PATCH 30/40 xen: move NUMA memory and CPU parsed nodemasks to common
In reply to Wei ChenBoth memory_nodes_parsed and processor_nodes_parsed are using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.
--- xen/arch/arm/numa_device_tree.c | 2 -- xen/arch/x86/srat.c | 3 --- xen/common/numa.c | 3 +++ xen/include/xen/nodemask.h | 2 ++ 4 files changed, 5 insertions(+), 5 deletions(-)
xen/arch/arm/numa_device_tree.c +0 -2
@@ -25,8 +25,6 @@#include <asm/setup.h> s8 device_tree_numa =0 ; -static nodemask_t processor_nodes_parsed__initdata ; -static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {
xen/arch/x86/srat.c +0 -3
@@ -24,9 +24,6 @@static struct acpi_table_slit *__read_mostly acpi_slit; -static nodemask_t memory_nodes_parsed__initdata ; -static nodemask_t processor_nodes_parsed__initdata ; -struct pxm2node {unsigned pxm;nodeid_t node;
xen/common/numa.c +3 -0
@@ -35,6 +35,9 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +nodemask_t memory_nodes_parsed__initdata ; +nodemask_t processor_nodes_parsed__initdata ; +bool numa_off;/*
xen/include/xen/nodemask.h +2 -0
@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits)*/ extern nodemask_t node_online_map; +extern nodemask_t memory_nodes_parsed; +extern nodemask_t processor_nodes_parsed;#if MAX_NUMNODES >1 #define num_online_nodes () nodes_weight(node_online_map)
Signature · 1 line
2.25.1
RFC PATCH 16/40 xen/arm: Create a fake NUMA node to use common code
In reply to Wei ChenWhen CONFIG_NUMA is enabled for Arm, Xen will switch to use common NUMA API instead of previous fake NUMA API. Before we parse NUMA information from device tree or ACPI SRAT table, we need to init the NUMA related variables, like cpu_to_node, as single node NUMA system.
So in this patch, we introduce a numa_init function for to initialize these data structures as all resources belongs to node#0. This will make the new API returns the same values as the fake API has done.
--- xen/arch/arm/numa.c | 53 ++++++++++++++++++++++++++++++++++++++ xen/arch/arm/setup.c | 8 ++++++ xen/include/asm-arm/numa.h | 11 ++++++++ 3 files changed, 72 insertions(+)
xen/arch/arm/numa.c +53 -0
@@ -20,6 +20,8 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/pfn.h> +#include <asm/setup.h> void numa_set_node (int cpu,nodeid_t nid) {
@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; } + +void __init numa_init (bool acpi_off) +{ +uint32_t idx; +paddr_t ram_start = ~0 ; +paddr_t ram_size =0 ; +paddr_t ram_end =0 ; + +printk (XENLOG_WARNING +"NUMA has not been supported yet, NUMA off!\n" ); +/* Arm NUMA has not been implemented until this patch */ + numa_off = true; + +/* +* Set all cpu_to_node mapping to 0, this will make cpu_to_node +* function return 0 as previous fake cpu_to_node API. +*/ +for ( idx =0 ; idx <NR_CPUS ; idx++ ) + cpu_to_node[idx] =0 ; + +/* +* Make node_to_cpumask, node_spanned_pages and node_start_pfn +* return as previous fake APIs. +*/ +for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { + node_to_cpumask[idx] = cpu_online_map; +node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); +node_start_pfn (idx) = (mfn_x (first_valid_mfn)); + } + +/* +* Find the minimal and maximum address of RAM, NUMA will +* build a memory to node mapping table for the whole range. +*/ + ram_start = bootinfo.mem.bank[0 ].start; + ram_size = bootinfo.mem.bank[0 ].size; + ram_end = ram_start + ram_size; +for ( idx =1 ; idx < bootinfo.mem.nr_banks; idx++ ) + { +paddr_t bank_start = bootinfo.mem.bank[idx].start; +paddr_t bank_size = bootinfo.mem.bank[idx].size; +paddr_t bank_end = bank_start + bank_size; + + ram_size = ram_size + bank_size; + ram_start =min (ram_start, bank_start); + ram_end =max (ram_end, bank_end); + } + +numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end)); +return ; +}
xen/arch/arm/setup.c +8 -0
@@ -30,6 +30,7 @@#include <xen/init.h> #include <xen/irq.h> #include <xen/mm.h> +#include <xen/numa.h> #include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h>
@@ -874,6 +875,13 @@ void __init start_xen(unsigned long boot_phys_offset,/* Parse the ACPI tables for possible boot-time configuration */ acpi_boot_table_init (); +/* +* Try to initialize NUMA system, if failed, the system will +* fallback to uniform system which means system has only 1 +* NUMA node. +*/ +numa_init (acpi_disabled); +end_boot_allocator ();/*
xen/include/asm-arm/numa.h +11 -0
@@ -13,6 +13,16 @@ typedef u8 nodeid_t;*/ #define NODES_SHIFT 6 +extern void numa_init (bool acpi_off); + +/* +* Temporary for fake NUMA node, when CPU, memory and distance +* matrix will be read from DTB or ACPI SRAT. The following +* symbols will be removed. +*/ +extern mfn_t first_valid_mfn; +#define __node_distance (a, b) (20) +#else /* Fake one node for now. See also node_online_map. */
@@ -35,6 +45,7 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_init (x) do { } while (0)#define numa_set_node (x, y) do { } while (0)#endif
Signature · 1 line
2.25.1
RFC PATCH 38/40 xen/arm: enable device tree based NUMA in system init
In reply to Wei ChenEverything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
--- xen/arch/arm/numa.c | 45 ++++++++++++++++++++++---------------- xen/include/asm-arm/numa.h | 7 ------ 2 files changed, 26 insertions(+), 26 deletions(-)
xen/arch/arm/numa.c +26 -19
@@ -18,6 +18,7 @@* */ #include <xen/init.h> +#include <xen/device_tree.h> #include <xen/nodemask.h> #include <xen/numa.h> #include <xen/pfn.h>
@@ -83,28 +84,34 @@ void __init numa_init(bool acpi_off)paddr_t ram_size =0 ;paddr_t ram_end =0 ; -printk (XENLOG_WARNING -"NUMA has not been supported yet, NUMA off!\n" ); -/* Arm NUMA has not been implemented until this patch */ - numa_off = true; +/* NUMA has been turned off through Xen parameters */ +if ( numa_off ) + goto mem_init; -/* -* Set all cpu_to_node mapping to 0, this will make cpu_to_node -* function return 0 as previous fake cpu_to_node API. -*/ -for ( idx =0 ; idx <NR_CPUS ; idx++ ) - cpu_to_node[idx] =0 ; - -/* -* Make node_to_cpumask, node_spanned_pages and node_start_pfn -* return as previous fake APIs. -*/ -for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { - node_to_cpumask[idx] = cpu_online_map; -node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); -node_start_pfn (idx) = (mfn_x (first_valid_mfn)); +/* Initialize NUMA from device tree when system is not ACPI booted */ +if ( acpi_off ) + { +#ifdef CONFIG_DEVICE_TREE_NUMA +int ret =numa_device_tree_init (device_tree_flattened); +if ( !ret ) + goto mem_init; +printk (XENLOG_WARNING +"Init NUMA from device tree failed, ret=%d\n" , ret); +#else +printk (XENLOG_WARNING +"CONFIG_DEVICE_TREE_NUMA is not set, NUMA off!\n" ); +#endif + numa_off = true; + } +else + { +/* We don't support NUMA for ACPI boot currently */ +printk (XENLOG_WARNING +"ACPI NUMA has not been supported yet, NUMA off!\n" ); + numa_off = true; } +mem_init:/* * Find the minimal and maximum address of RAM, NUMA will * build a memory to node mapping table for the whole range.
xen/include/asm-arm/numa.h +0 -7
@@ -30,13 +30,6 @@ extern int numa_device_tree_init(const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);extern void arch_numa_init_failed_fallback (void ); -/* -* Temporary for fake NUMA node, when CPU, memory and distance -* matrix will be read from DTB or ACPI SRAT. The following -* symbols will be removed. -*/ -extern mfn_t first_valid_mfn; -#else /* Fake one node for now. See also node_online_map. */
Signature · 1 line
2.25.1
RFC PATCH 29/40 xen/arm: implement Arm arch helpers Arm to get memory map info
In reply to Wei ChenThese two helpers are architecture APIs that are required by nodes_cover_memory.
--- xen/arch/arm/numa.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
xen/arch/arm/numa.c +14 -0
@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end));return ; } + +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return bootinfo.mem.nr_banks; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end) +{ + *start = bootinfo.mem.bank[bank].start; + *end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size; + +return 0 ; +}
Signature · 1 line
2.25.1
RFC PATCH 33/40 xen: export bad_srat and srat_disabled to extern
In reply to Wei ChenThese two functions are architecture implementation. But common code out of arch will invoke them, so we export them to extern.
--- xen/arch/arm/numa_device_tree.c | 4 ++-- xen/arch/x86/srat.c | 2 +- xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 ++ 4 files changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/numa_device_tree.c +2 -2
@@ -26,12 +26,12 @@s8 device_tree_numa =0 ; -static int srat_disabled (void ) +int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -static __init void bad_srat (void ) +__init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ;
xen/arch/x86/srat.c +1 -1
@@ -97,7 +97,7 @@ nodeid_t setup_node(unsigned pxm)return node; } -static __init void bad_srat (void ) +__init void bad_srat (void ) {int i;printk (KERN_ERR "SRAT: SRAT not used.\n" );
xen/include/asm-x86/numa.h +0 -1
@@ -13,7 +13,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern int srat_disabled (void );extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);
xen/include/xen/numa.h +2 -0
@@ -75,6 +75,8 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,extern int conflicting_memblks (u64 start,u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern int srat_disabled (void ); +extern void bad_srat (void );extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);
Signature · 1 line
2.25.1
RFC PATCH 18/40 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Wei ChenEFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to parse memory nodes in EFI boot.
As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +7 -1
@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ; -if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, +* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) ) rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem);else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )
xen/arch/arm/efi/efi-boot.h +0 -25
@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions.
xen/include/asm-arm/setup.h +6 -0
@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32 address_cells,u32 device_tree_get_u32 (const void *fdt,int node,const char *prop_name,u32 dflt); +#if defined(CONFIG_ARM_64 ) +#define arch_efi_enabled (x) efi_enabled(x) +#else +#define arch_efi_enabled (x) (0) +#endif +#endif /* * Local variables:
Signature · 1 line
2.25.1
RFC PATCH 24/40 xen/arm: introduce a helper to parse device tree NUMA distance map
In reply to Wei ChenA NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
xen/arch/arm/numa_device_tree.c +67 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i; + +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + } + +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 ) + { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++; + to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + +if ( (from == to && distance !=NUMA_LOCAL_DISTANCE ) || + (from != to && distance <=NUMA_LOCAL_DISTANCE ) ) + { +printk (XENLOG_WARNING +"Invalid nodes' distance from node#%d to node#%d = %d\n" , + from, to, distance); +return -EINVAL ; + } + +printk (XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n" , + from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance); + } + +return 0 ; +}
Signature · 1 line
2.25.1
RFC PATCH 23/40 xen/arm: introduce a helper to parse device tree memory node
In reply to Wei ChenMemory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)
xen/arch/arm/numa_device_tree.c +130 -0
@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {
@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; + + end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */ + i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) { +/* +* Overlaps with other memblk in the same node, warning here. +* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 +"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , + node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++; + +return 0 ; +} +/* Parse CPU NUMA node info */ int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) {
@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)return dtb_numa_processor_affinity_init (nid); } + +/* Parse memory node NUMA info */ +int __init +device_tree_parse_numa_memory_node (const void *fdt,int node, +const char *name,uint32_t addr_cells,uint32_t size_cells) +{ +uint32_t nid; +int ret =0 , len; +paddr_t addr, size; +const struct fdt_property *prop; +uint32_t idx, ranges; +const __be32 *addresses; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + + prop =fdt_get_property (fdt, node,"reg" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"fdt: node `%s': missing `reg' property\n" , name); +return -EINVAL ; + } + + addresses = (const __be32 *)prop->data; + ranges = len / (sizeof (__be32)* (addr_cells + size_cells)); +for ( idx =0 ; idx < ranges; idx++ ) + { +device_tree_get_reg (&addresses, addr_cells, size_cells, &addr, &size); +/* Skip zero size ranges */ +if ( !size ) +continue ; + + ret =dtb_numa_memory_affinity_init (nid, addr, size); +if ( ret ) { +printk (XENLOG_WARNING +"NUMA: process range#%d addr = %lx size=%lx failed!\n" , + idx, addr, size); +return -EINVAL ; + } + } + +if ( idx ==0 ) + { +printk (XENLOG_ERR +"bad property in memory node, idx=%d ret=%d\n" , idx, ret); +return -EINVAL ; + } + +return 0 ; +}
Signature · 1 line
2.25.1
RFC PATCH 21/40 xen/arm: introduce device_tree_numa as a switch for device tree NUMA
In reply to Wei ChenLike acpi_numa in x86 as a switch for ACPI based NUMA, we introduce device_tree_numa as a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.
Keep using bad_srat and srat_disabled functions name, because we will reuse node_covers_memory and acpi_scan_nodes code for Arm. These functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.
--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ 3 files changed, 38 insertions(+) create mode 100644 xen/arch/arm/numa_device_tree.c
xen/arch/arm/Makefile +1 -0
@@ -36,6 +36,7 @@ obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.o
xen/arch/arm/numa_device_tree.c new file +35 -0
@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +s8 device_tree_numa =0 ; + +int srat_disabled (void ) +{ +return numa_off || device_tree_numa <0 ; +} + +void __init bad_srat (void ) +{ +printk (KERN_ERR "DT: NUMA information is not used.\n" ); + device_tree_numa =-1 ; +}
xen/include/asm-arm/numa.h +2 -0
@@ -23,6 +23,8 @@ typedef u8 nodeid_t;#define NUMA_LOCAL_DISTANCE 10#define NUMA_REMOTE_DISTANCE 20 +extern s8 device_tree_numa; +extern void numa_init (bool acpi_off);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);
Signature · 1 line
2.25.1
RFC PATCH 28/40 xen/x86: decouple nodes_cover_memory with E820 map
In reply to Wei ChenWe will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range
Depends above two helpers, we make nodes_cover_memory become architecture independent.
--- xen/arch/x86/numa.c | 18 ++++++++++++++++++ xen/arch/x86/srat.c | 8 +++----- xen/include/xen/numa.h | 4 ++++ 3 files changed, 25 insertions(+), 5 deletions(-)
xen/arch/x86/numa.c +18 -0
@@ -128,6 +128,24 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) { +return -1 ; + } + + *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();
xen/arch/x86/srat.c +3 -5
@@ -316,18 +316,16 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)static int __init nodes_cover_memory (void ) {int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); -for (i =0 ; i < e820.nr_map; i++) { +for (i =0 ; i < nr_banks; i++) {int j, found;unsigned long long start, end; -if (e820.map[i].type !=E820_RAM ) { +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) {continue ; } - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; - do { found = 0; for_each_node_mask(j, memory_nodes_parsed)
xen/include/xen/numa.h +4 -0
@@ -89,6 +89,10 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } +extern uint32_t arch_meminfo_get_nr_bank (void ); +extern int arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
RFC PATCH 31/40 xen/x86: move nodes_cover_memory to common
In reply to Wei ChenNot only ACPU NUMA, but also Arm device tree based NUMA will use nodes_cover_memory to do sanity check. So we move this function from arch/x86 to common.
--- xen/arch/x86/srat.c | 40 ---------------------------------------- xen/common/numa.c | 40 ++++++++++++++++++++++++++++++++++++++++ xen/include/xen/numa.h | 1 + 3 files changed, 41 insertions(+), 40 deletions(-)
xen/arch/x86/srat.c +0 -40
@@ -308,46 +308,6 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) num_node_memblks++; } -/* Sanity check to catch more bad SRATs (they are amazingly common). -Make sure the PXMs cover all memory. */ -static int __init nodes_cover_memory (void ) -{ -int i; -uint32_t nr_banks =arch_meminfo_get_nr_bank (); - -for (i =0 ; i < nr_banks; i++) { -int j, found; -unsigned long long start, end; - -if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { -continue ; - } - -do { - found =0 ; -for_each_node_mask (j, memory_nodes_parsed) -if (start < nodes[j].end - && end > nodes[j].start) { -if (start >= nodes[j].start) { - start = nodes[j].end; - found =1 ; - } -if (end <= nodes[j].end) { - end = nodes[j].start; - found =1 ; - } - } - }while (found && start < end); - -if (start < end) { -printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); -return 0 ; - } - } -return 1 ; -} -void __init acpi_numa_arch_fixup (void ) {}static uint64_t __initdata srat_region_mask;
xen/common/numa.c +40 -0
@@ -193,6 +193,46 @@ void __init cutoff_node(int i, u64 start, u64 end) } } +/* Sanity check to catch more bad SRATs (they are amazingly common). +Make sure the PXMs cover all memory. */ +int __init nodes_cover_memory (void ) +{ +int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); + +for (i =0 ; i < nr_banks; i++) { +int j, found; +unsigned long long start, end; + +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { +continue ; + } + +do { + found =0 ; +for_each_node_mask (j, memory_nodes_parsed) +if (start < nodes[j].end + && end > nodes[j].start) { +if (start >= nodes[j].start) { + start = nodes[j].end; + found =1 ; + } +if (end <= nodes[j].end) { + end = nodes[j].start; + found =1 ; + } + } + }while (found && start < end); + +if (start < end) { +printk (KERN_ERR "SRAT: No PXM for e820 range: " +"%016Lx - %016Lx\n" , start, end); +return 0 ; + } + } +return 1 ; +} +void numa_add_cpu (int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]);
xen/include/xen/numa.h +1 -0
@@ -92,6 +92,7 @@ static inline void clear_node_cpumask(int cpu)extern uint32_t arch_meminfo_get_nr_bank (void );extern int arch_meminfo_get_ram_bank_range (int bank,unsigned long long *start,unsigned long long *end); +extern int nodes_cover_memory (void );#endif /* CONFIG_NUMA */
Signature · 1 line
2.25.1
RFC PATCH 19/40 xen: fdt: Introduce a helper to check fdt node type
In reply to Wei ChenIn later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce fdt_node_check_type in this patch to avoid redundant code in subsequent patches.
--- xen/common/libfdt/fdt_ro.c | 15 +++++++++++++++ xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
xen/common/libfdt/fdt_ro.c +15 -0
@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, int nodeoffset,return 1 ; } +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type) +{ +const void *prop; +int len; + + prop =fdt_getprop (fdt, nodeoffset,"device_type" , &len); +if (!prop) +return len; +if (fdt_stringlist_contains (prop, len, type)) +return 0 ; +else +return 1 ; +} +int fdt_node_offset_by_compatible (const void *fdt,int startoffset,const char *compatible) {
xen/include/xen/libfdt/libfdt.h +25 -0
@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);int fdt_node_check_compatible (const void *fdt,int nodeoffset,const char *compatible); +/** +* fdt_node_check_type: check a node's device_type property +* @fdt: pointer to the device tree blob +* @nodeoffset: offset of a tree node +* @type: string to match against +* +* +* fdt_node_check_type() returns 0 if the given node contains a 'device_type' +* property with the given string as one of its elements, it returns non-zero +* otherwise, or on error. +* +* returns: +* 0, if the node has a 'device_type' property listing the given string +* 1, if the node has a 'device_type' property, but it does not list +* the given string +* -FDT_ERR_NOTFOUND, if the given node has no 'device_type' property +* -FDT_ERR_BADOFFSET, if nodeoffset does not refer to a BEGIN_NODE tag +* -FDT_ERR_BADMAGIC, +* -FDT_ERR_BADVERSION, +* -FDT_ERR_BADSTATE, +* -FDT_ERR_BADSTRUCTURE, standard meanings +*/ +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type); +/** * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value * @fdt: pointer to the device tree blob
Signature · 1 line
2.25.1
RFC PATCH 40/40 xen/x86: move dump_numa info hotkey to common
In reply to Wei ChenAs device tree based NUMA has been enabled for Arm, so not only x86 needs to dump numa info through hotkey, but also Arm can use this hotkey to dump numa info.
In this patch, we move this hotkey to common. Arm can use it to dump its numa information:
(XEN) key 'u' (ascii '75') => dump NUMA info (XEN) 'u' pressed -> dumping numa info (now = 8805901249990) (XEN) NODE0 start->524288 size->520192 free->257673 (XEN) NODE1 start->8912896 size->524288 free->499676 (XEN) CPU0...1 -> NODE0
(XEN) CPU2...3 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 262144):
(XEN) Node 0: 262144
(XEN) Node 1: 0
--- xen/arch/x86/numa.c | 142 ------------------------------------------- xen/common/numa.c | 144 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+), 142 deletions(-)
xen/arch/x86/numa.c +0 -142
@@ -112,145 +112,3 @@ int __init arch_meminfo_get_ram_bank_range(int bank,return 0 ; } - -static void dump_numa (unsigned char key) -{ -s_time_t now =NOW (); -unsigned int i, j, n; -struct domain *d; -struct page_info *page; -unsigned int page_num_node[MAX_NUMNODES ]; -const struct vnuma_info *vnuma; - -printk ("'%c' pressed -> dumping numa info (now = %" PRI_stime")\n" , key, - now); - -for_each_online_node ( i ) - { -paddr_t pa =pfn_to_paddr (node_start_pfn (i) +1 ); - -printk ("NODE%u start->%lu size->%lu free->%lu\n" , - i,node_start_pfn (i),node_spanned_pages (i), -avail_node_heap_pages (i)); -/* sanity check phys_to_nid() */ -if (phys_to_nid (pa) != i ) -printk ("phys_to_nid(%" PRIpaddr") -> %d should be %u\n" , - pa,phys_to_nid (pa), i); - } - - j =cpumask_first (&cpu_online_map); - n =0 ; -for_each_online_cpu ( i ) - { -if ( i != j + n || cpu_to_node[j] != cpu_to_node[i] ) - { -if ( n >1 ) -printk ("CPU%u...%u -> NODE%d\n" , j, j + n -1 , cpu_to_node[j]); -else -printk ("CPU%u -> NODE%d\n" , j, cpu_to_node[j]); - j = i; - n =1 ; - } -else - ++n; - } -if ( n >1 ) -printk ("CPU%u...%u -> NODE%d\n" , j, j + n -1 , cpu_to_node[j]); -else -printk ("CPU%u -> NODE%d\n" , j, cpu_to_node[j]); - -rcu_read_lock (&domlist_read_lock); - -printk ("Memory location of each domain:\n" ); -for_each_domain ( d ) - { -process_pending_softirqs (); - -printk ("Domain %u (total: %u):\n" , d->domain_id,domain_tot_pages (d)); - -for_each_online_node ( i ) - page_num_node[i] =0 ; - -spin_lock (&d->page_alloc_lock); -page_list_for_each (page, &d->page_list) - { - i =phys_to_nid (page_to_maddr (page)); - page_num_node[i]++; - } -spin_unlock (&d->page_alloc_lock); - - for_each_online_node ( i ) - printk(" Node %u: %u\n", i, page_num_node[i]); - -if ( !read_trylock (&d->vnuma_rwlock) ) -continue ; - -if ( !d->vnuma ) - { -read_unlock (&d->vnuma_rwlock); -continue ; - } - - vnuma = d->vnuma; -printk (" %u vnodes, %u vcpus, guest physical layout:\n" , - vnuma->nr_vnodes, d->max_vcpus); -for ( i =0 ; i < vnuma->nr_vnodes; i++ ) - { -unsigned int start_cpu = ~0U ; - -if ( vnuma->vnode_to_pnode[i] ==NUMA_NO_NODE ) -printk (" %3u: pnode ???," , i); -else -printk (" %3u: pnode %3u," , i, vnuma->vnode_to_pnode[i]); - -printk (" vcpus " ); - -for ( j =0 ; j < d->max_vcpus; j++ ) - { -if ( !(j &0x3f ) ) -process_pending_softirqs (); - -if ( vnuma->vcpu_to_vnode[j] == i ) - { -if ( start_cpu == ~0U ) - { -printk ("%d" , j); - start_cpu = j; - } - } -else if ( start_cpu != ~0U ) - { -if ( j -1 != start_cpu ) -printk ("-%d " , j -1 ); -else -printk (" " ); - start_cpu = ~0U ; - } - } - -if ( start_cpu != ~0U && start_cpu != j -1 ) -printk ("-%d" , j -1 ); - -printk ("\n" ); - -for ( j =0 ; j < vnuma->nr_vmemranges; j++ ) - { -if ( vnuma->vmemrange[j].nid == i ) -printk (" %016" PRIx64" - %016" PRIx64"\n" , - vnuma->vmemrange[j].start, - vnuma->vmemrange[j].end); - } - } - -read_unlock (&d->vnuma_rwlock); - } - -rcu_read_unlock (&domlist_read_lock); -} - -static __init int register_numa_trigger (void ) -{ -register_keyhandler ('u' , dump_numa,"dump NUMA info" ,1 ); -return 0 ; -} -__initcall (register_numa_trigger);
xen/common/numa.c +144 -0
154 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH 27/40 xen/arm: build CPU NUMA node map while creating cpu_logical_map
In reply to Wei ChenSometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.
--- xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
xen/arch/arm/smpboot.c +30 -1
@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void) { [0 ... NR_CPUS - 1] = MPIDR_INVALID }; +static nodeid_t node_map[NR_CPUS ]__initdata = + { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE + };bool bootcpu_valid = false; +uint32_t nid =0 ;int rc; mpidr = boot_cpu_data.mpidr.bits &MPIDR_HWID_MASK ;
@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)continue ; } +#ifdef CONFIG_DEVICE_TREE_NUMA +/* +* When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation +* from CPU dts node, otherwise the nid is always 0. +*/ +if ( !dt_property_read_u32 (cpu,"numa-node-id" , &nid) ) + { +printk (XENLOG_WARNING +"cpu[%d] dts path: %s: doesn't have numa infomation!\n" , + cpuidx,dt_node_full_name (cpu)); +/* +* The the early stage of NUMA initialization, when Xen found any +* CPU dts node doesn't have numa-node-id info, the NUMA will be +* treated as off, all CPU will be set to a FAKE node 0. So if we +* get numa-node-id failed here, we should set nid to 0. +*/ + nid =0 ; + } +#endif +/* * 8 MSBs must be set to 0 in the DT since the reg property * defines the MPIDR[23:0]
@@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void) {printk ("cpu%d init failed (hwid %" PRIregister"): %d\n" , i, hwid, rc); tmp_map[i] =MPIDR_INVALID ; + node_map[i] =NUMA_NO_NODE ; } - else + else { tmp_map[i] = hwid; + node_map[i] = nid; + } }if ( !bootcpu_valid )
@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)continue ;cpumask_set_cpu (i, &cpu_possible_map);cpu_logical_map (i) = tmp_map[i]; +numa_set_node (i, node_map[i]); } }
Signature · 1 line
2.25.1
RFC PATCH 22/40 xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenProcessor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)
xen/arch/arm/numa_device_tree.c +39 -2
@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Signature · 1 line
2.25.1
RFC PATCH 15/40 xen/x86: move NUMA API from x86 header to common header
In reply to Wei ChenAs all functions and macros that depends on these NUMA API, like clear_node_cpumask, have been moved to common header file. In this case, we can move NUMA API from x86 header file to common header file, and will not trigger the symbols not found error when functions in arch NUMA header file depends on the symbols in common NUMA header file.
--- xen/include/asm-x86/numa.h | 13 ------------- xen/include/xen/numa.h | 13 +++++++++++++ 2 files changed, 13 insertions(+), 13 deletions(-)
xen/include/asm-x86/numa.h +0 -13
@@ -9,14 +9,6 @@ typedef u8 nodeid_t;extern int srat_rev; -extern nodeid_t cpu_to_node[NR_CPUS ]; -extern cpumask_t node_to_cpumask[]; - -#define cpu_to_node (cpu) (cpu_to_node[cpu]) -#define parent_node (node) (node) -#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node])) -#define node_to_cpumask (node) (node_to_cpumask[node]) -extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -28,11 +20,6 @@ extern void srat_detect_node(int cpu);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void ); -static inline void clear_node_cpumask (int cpu) -{ -cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); -} -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );
xen/include/xen/numa.h +13 -0
@@ -22,6 +22,14 @@/* The following content can be used when NUMA feature is enabled */ #if defined(CONFIG_NUMA ) +extern nodeid_t cpu_to_node[NR_CPUS ]; +extern cpumask_t node_to_cpumask[]; + +#define cpu_to_node (cpu) (cpu_to_node[cpu]) +#define parent_node (node) (node) +#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node])) +#define node_to_cpumask (node) (node_to_cpumask[node]) +/* Simple perfect hash to map pdx to node numbers */ extern int memnode_shift;extern unsigned long memnodemapsize;
@@ -76,6 +84,11 @@ extern int numa_fake;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +static inline void clear_node_cpumask (int cpu) +{ +cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); +} +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
RFC PATCH 13/40 xen/arm: introduce numa_set_node for Arm
In reply to Wei ChenThis API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed, node_online_map will be cleared and set node#0 online. So we use node_online_map to prevent to set a CPU to an offline node.
--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa.c | 31 +++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-) create mode 100644 xen/arch/arm/numa.c
xen/arch/arm/Makefile +1 -0
@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o +obj-$(CONFIG_NUMA) += numa.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.o
xen/arch/arm/numa.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +void numa_set_node (int cpu,nodeid_t nid) +{ +if ( nid >=MAX_NUMNODES || + !nodemask_test (nid, &node_online_map) ) + nid =0 ; + + cpu_to_node[cpu] = nid; +}
xen/include/asm-arm/numa.h +2 -0
@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_set_node (x, y) do { } while (0) +#endif #endif /* __ARCH_ARM_NUMA_H */
xen/include/asm-x86/numa.h +0 -1
@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))extern int srat_disabled (void ); -extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);
xen/include/xen/numa.h +1 -0
@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern void numa_set_node (int cpu,nodeid_t node);extern bool numa_off;extern int numa_fake;
Signature · 1 line
2.25.1
RFC PATCH 32/40 xen/x86: make acpi_scan_nodes to be neutral
In reply to Wei ChenThe code in acpi_scan_nodes can be reused for device tree based NUMA. So we rename acpi_scan_nodes to numa_scan_nodes for a neutral function name. As acpi_numa variable is available in ACPU based NUMA system only, we use CONFIG_ACPI_NUMA to protect it.
--- xen/arch/x86/srat.c | 4 +++- xen/common/numa.c | 2 +- xen/include/asm-x86/acpi.h | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)
xen/arch/x86/srat.c +3 -1
@@ -362,7 +362,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (u64 start,u64 end) +int __init numa_scan_nodes (u64 start,u64 end) {int i;nodemask_t all_nodes_parsed;
@@ -371,8 +371,10 @@ int __init acpi_scan_nodes(u64 start, u64 end)for (i =0 ; i <MAX_NUMNODES ; i++)cutoff_node (i, start, end); +#ifdef CONFIG_ACPI_NUMA if (acpi_numa <=0 )return -1 ; +#endif if (!nodes_cover_memory ()) {bad_srat ();
xen/common/numa.c +1 -1
@@ -330,7 +330,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)#endif #ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , +if ( !numa_off && !numa_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ) )return ;#endif
xen/include/asm-x86/acpi.h +1 -1
@@ -102,7 +102,7 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1extern s8 acpi_numa; -extern int acpi_scan_nodes (u64 start,u64 end); +extern int numa_scan_nodes (u64 start,u64 end);extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)
Signature · 1 line
2.25.1
RFC PATCH 9/40 xen/x86: Move numa_add_cpu_node to common
In reply to Wei ChenThis function will be reused by Arm later, so we move it from arch/x86 to common. But we keep cpu_to_node and node_to_cpumask to x86 header file. Because cpu_to_node and node_to_cpumask have different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.
--- xen/arch/x86/numa.c | 9 --------- xen/common/numa.c | 11 +++++++++++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 ++ 4 files changed, 13 insertions(+), 10 deletions(-)
xen/arch/x86/numa.c +0 -9
@@ -29,16 +29,12 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { - [0 ...NR_CPUS -1 ] =NUMA_NO_NODE -};/* * Keep BIOS's CPU2node information, should not be used for memory allocaion */ nodeid_t apicid_to_node[MAX_LOCAL_APIC ] = { [0 ...MAX_LOCAL_APIC -1 ] =NUMA_NO_NODE }; -cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ;nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } };
@@ -167,11 +163,6 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) (u64)end_pfn << PAGE_SHIFT); } -void numa_add_cpu (int cpu) -{ -cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;
xen/common/numa.c +11 -0
@@ -23,6 +23,12 @@ typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap; +nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE +}; + +cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ; +/* * Given a shift value, try to populate memnodemap[] * Returns :
@@ -129,3 +135,8 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,return shift; } + +void numa_add_cpu (int cpu) +{ +cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); +}
xen/include/asm-x86/numa.h +0 -1
@@ -27,7 +27,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_add_cpu (int cpu);extern void numa_init_array (void );extern bool numa_off;
xen/include/xen/numa.h +2 -0
@@ -51,6 +51,8 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) +extern void numa_add_cpu (int cpu); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
RFC PATCH 36/40 xen/arm: keep guest still be NUMA unware
In reply to Wei ChenWe have not wanted to make Xen guest be NUMA aware in this patch series. So in this patch, Xen will skip NUMA distance matrix node and skip the numa-node-id property in CPU node and memory node, when Xen is creating guest device tree binary.
--- xen/arch/arm/domain_build.c | 6 ++++++ 1 file changed, 6 insertions(+)
xen/arch/arm/domain_build.c +6 -0
@@ -584,6 +584,10 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,continue ; } +/* Guest is numa unaware in current stage */ +if (dt_property_name_is_equal (prop,"numa-node-id" ) ) +continue ; + res =fdt_property (kinfo->fdt, prop->name, prop_data, prop_len);if ( res )
@@ -1454,6 +1458,8 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,DT_MATCH_TYPE ("memory" ),/* The memory mapped timer is not supported by Xen. */ DT_MATCH_COMPATIBLE ("arm,armv7-timer-mem" ), +/* Numa info doesn't need to be exposed to Domain-0 */ + DT_MATCH_COMPATIBLE("numa-distance-map-v1"), { /* sentinel */ }, }; static const struct dt_device_match timer_matches[] __initconst =
Signature · 1 line
2.25.1
RFC PATCH 26/40 xen/arm: Add boot and secondary CPU to NUMA system
In reply to Wei ChenWhen cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.
--- xen/arch/arm/setup.c | 6 ++++++ xen/arch/arm/smpboot.c | 6 ++++++ xen/include/asm-arm/numa.h | 1 + 3 files changed, 13 insertions(+)
xen/arch/arm/setup.c +6 -0
@@ -918,6 +918,12 @@ void __init start_xen(unsigned long boot_phys_offset,processor_id (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (0 ); +smp_init_cpus (); cpus =smp_get_max_cpus ();printk (XENLOG_INFO "SMP: Allowing %u CPUs\n" , cpus);
xen/arch/arm/smpboot.c +6 -0
@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +/* Now report this CPU is up */ cpumask_set_cpu (cpuid, &cpu_online_map);
xen/include/asm-arm/numa.h +1 -0
@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;#define __node_distance (a, b) (20)#define numa_init (x) do { } while (0) +#define numa_add_cpu (x) do { } while (0)#define numa_set_node (x, y) do { } while (0)#endif
Signature · 1 line
2.25.1
Quoted 36 lines · Wei Chen
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto extern
Quoted 8 lines
xen: move
numa_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to common
May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.
Thanks and regards,
Jan
Re: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
In reply to Wei ChenQuoted 4 lines · Wei Chen
| libxlu_pci.c: In function 'xlu_pci_parse_bdf': | libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 32 | pcidev->func = func; | | ~~~~~~~~~~~~~^~~~~~
I'm afraid I can't spot such an assignment in the file (nor the two similar ones further down). All I can see is
pci->domain = domain;
pci->bus = bus;
pci->dev = dev;
pci->func = func;Quoted 10 lines
| libxlu_pci.c:51:29: note: 'func' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~~ | libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 31 | pcidev->dev = dev; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:24: note: 'dev' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this function [-Werror=maybe-uninitialized]
Quoted 8 lines
| 30 | pcidev->bus = bus; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:19: note: 'bus' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 78 | if ( dom & ~0xff ) | | ~~~~^~~~~~~
I'm afraid I also can't spot a variable named "dom", nor a sufficiently similar if(). May I ask what code base these were observed with? Is the change needed at all anymore?
Quoted 13 lines
tools/libs/util/libxlu_pci.c +1
-1@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) {const char *ptr = str;unsigned int colons =0 ; -unsigned int domain, bus, dev, func; +unsigned int domain =0 , bus =0 , dev =0 , func =0 ;int n;/* Count occurrences of ':' to detrmine presence/absence of the 'domain' */
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) · line 33
Quoted 7 lines
tools/libs/util/libxlu_pci.c +0
-1@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) ptr = str;switch (colons) {case 1 : - domain =0 ;if (sscanf (ptr,"%x:%x.%n" , &bus, &dev, &n) !=2 )return ERROR_INVAL ;break ;
Also - which compiler did you encounter this with?
Finally please don't forget to Cc maintainers.
Jan
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Wei Chenxen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30
Quoted 7 lines · Wei Chen
xen/arch/arm/platform.c +1
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;
This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230
Quoted 8 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;
... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.
Also I'm afraid the description does not make clear why this variable is what you want to use. Connected to this is the question why you need to consume it on Arm in the first place, when x86 never had the need.
Jan
Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node
In reply to Wei ChenQuoted 10 lines · Wei Chen
When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of
_memnodemap. Xen hadn't done it, and Xen should assert inphys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.
How about we promote VIRTUAL_BUG_ON() to expand to at least ASSERT()?
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273
Quoted 7 lines
xen/arch/x86/numa.c +1
-0@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap);
But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.
Jan
Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
In reply to Wei Chenxen/arch/x86/Kconfig config X86 · line 27
Quoted 8 lines · Wei Chen
xen/arch/x86/Kconfig +1
-1@@ -24,7 +24,7 @@ config X86 select HAS_UBSAN select HAS_VPCI if HVM select NEEDS_LIBELF - select NUMA + select ACPI_NUMA
We try to keep this alphabetically sorted, so please move up the replacement line. Then
Jan
Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
In reply to Wei ChenHi Wei,
Quoted 1 line · Wei Chen
In current Xen code only implments x86 ACPI-based NUMA support.
s/implments/implements/
Quoted 5 lines
So in Xen
Kconfigsystem, NUMA equals toACPI_NUMA. x86 selects NUMA by default, andCONFIG_ACPI_NUMAis hardcode inconfig.h.In this patch series, we introduced device tree based NUMA for Arm.
The concept of patch series only applies to the ML. Once checked-in this is only a series of commit. So I would write:
"In a follow-up patch, we will introduce support for NUMA using the Device-Tree".
Quoted 1 line
That means we will have two NUMA implemetations, so in this
s/implemetations/implementations/
Quoted 2 lines
patch we decouple NUMA from ACPI based NUMA in
Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.
Cheers,
Signature · 1 line
Julien Grall
Hi Jan,
Quoted 55 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:42
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64Quoted 36 lines · Wei Chen
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto externQuoted 8 lines
xen: move
numa_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to common
Quoted 4 lines
May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.
Thanks for your reminder. Before I sent this series, I had paid sometime to consider CC or TO you, I found you are in the X86 Arch, x86 memory management maintainer lists. And in this patch series, I have done some changes that affects x86, so I added you in TO list. Obviously, my understanding had some mistake. I will add you to CC list in next version.
Quoted 2 lines
Thanks and regards,
Jan
RE: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
In reply to Jan BeulichHi Jan,
Quoted 16 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:50
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfQuoted 2 lines · Wei Chen
|
libxlu_pci.c: In function 'xlu_pci_parse_bdf':
| libxlu_pci.c:32:18: error: 'func' may be used uninitialized in thisfunction [-Werror=maybe-uninitialized]
Quoted 2 lines
| 32 | pcidev->func = func; | | ~~~~~~~~~~~~~^~~~~~
Quoted 8 lines
I'm afraid I can't spot such an assignment in the file (nor the two similar ones further down). All I can see is
pci->domain = domain; pci->bus = bus; pci->dev = dev; pci->func = func;
Sorry, I forgot to update my commit log with the latest code base. I revert this change in my current code, I can't reproduce it. I'm not sure if it's because I upgraded my build environment. Give me sometime, if I can reproduce it I will update the commit log in next version. If it's no longer needed, I will remove this patch from this series.
Quoted 18 lines
Quoted 4 lines
| libxlu_pci.c:51:29: note: 'func' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~~ | libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in thisfunction [-Werror=maybe-uninitialized]
Quoted 6 lines
| 31 | pcidev->dev = dev; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:24: note: 'dev' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in thisfunction [-Werror=maybe-uninitialized]
Quoted 6 lines
| 30 | pcidev->bus = bus; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:19: note: 'bus' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this
Quoted 8 lines
function [-Werror=maybe-uninitialized]
Quoted 2 lines
| 78 | if ( dom & ~0xff ) | | ~~~~^~~~~~~I'm afraid I also can't spot a variable named "dom", nor a sufficiently similar
if(). May I ask what code base these were observed with? Is the change needed at all anymore?
same as above.
Quoted 24 lines
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char
Quoted 3 lines
tools/libs/util/libxlu_pci.c +0
-0@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char*str, const char **endp)
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp)
Quoted 8 lines
tools/libs/util/libxlu_pci.c +1
-1{const char *ptr = str;unsigned int colons =0 ; -unsigned int domain, bus, dev, func; +unsigned int domain =0 , bus =0 , dev =0 , func =0 ;int n;/* Count occurrences of ':' to detrmine presence/absence of the 'domain' */
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char
Quoted 1 line
tools/libs/util/libxlu_pci.c +0
-0@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char*str, const char **endp)
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) · line 33
Quoted 8 lines
tools/libs/util/libxlu_pci.c +0
-1ptr = str;switch (colons) {case 1 : - domain =0 ;if (sscanf (ptr,"%x:%x.%n" , &bus, &dev, &n) !=2 )return ERROR_INVAL ;break ;
Quoted 4 lines
Also - which compiler did you encounter this with?
Finally please don't forget to Cc maintainers.
If this patch still needed, I will do in next version.
Quoted 1 line
Jan
Quoted 54 lines · Wei Chen
Quoted 49 lines
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:42Quoted 36 lines · Wei Chen
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto externQuoted 8 lines
xen: move
numa_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to commonQuoted 4 lines
May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.
Quoted 6 lines
Thanks for your reminder. Before I sent this series, I had paid sometime to consider CC or TO you, I found you are in the X86 Arch, x86 memory management maintainer lists. And in this patch series, I have done some changes that affects x86, so I added you in TO list. Obviously, my understanding had some mistake. I will add you to CC list in next version.
And then on a patch-by-patch basis please, unless you see a specific need to also Cc my on certain Arm-only patches. Thanks.
Jan
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Jan BeulichHi Jan,
Quoted 14 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:54
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setxen/arch/arm/platform.c extern const struct platform_desc _splatform[],
Quoted 3 lines · Wei Chen
xen/arch/arm/platform.c +0
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],_eplatform[];
Quoted 7 lines
xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30
Quoted 4 lines
xen/arch/arm/platform.c +1
-0/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...
Ok, I will place it to a header.
Quoted 13 lines
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230
Quoted 9 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.
Sorry, I am not very clear about this comment.
Quoted 4 lines
Also I'm afraid the description does not make clear why this variable is what you want to use. Connected to this is the question why you need to consume it on Arm in the first place, when x86 never had the need.
Different Arm platforms may have different DMA bitsize. So in my previous thought, If Arm platform doesn't provide any DMA bitsize info, I will return the system DMA bitsize (dma_bitsize) in arch_get_dma_bitsize. But your comment made me think again. My current change may have fallen into a logical loophole. dma_bitsize as a high level variable, it's value depends on xen boot command line or arch_get_dma_bitsize. I can't use it for arch_get_dma_bitsize's input.
So I think, in next version, I will discard the changes of dma_bitsize, just keep the changes in arch_get_dma_bitsize to return 0, when platform hasn't specify the DMA bitsize. Just like:
unsigned int arch_get_dma_bitsize(void)
{
- return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
+ return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize
+ : 0;
}
Thanks,
Wei Chen
Quoted 1 line
Jan
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Quoted 32 lines · Wei Chen
Quoted 8 lines
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:54xen/arch/arm/platform.c extern const struct platform_desc _splatform[],
Quoted 3 lines · Wei Chen
xen/arch/arm/platform.c +0
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],_eplatform[];
Quoted 7 lines
xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30
Quoted 4 lines
xen/arch/arm/platform.c +1
-0/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...
Ok, I will place it to a header.
Quoted 13 lines
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230
Quoted 9 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.
Quoted 1 line
Sorry, I am not very clear about this comment.
I've merely been trying to explain _why_ the declaration needs to be in a header.
Jan
RE: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node
In reply to Jan BeulichHi Jan,
Quoted 14 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:33
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA nodeQuoted 4 lines · Wei Chen
When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
Quoted 9 lines
Quoted 6 lines
In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of
_memnodemap. Xen hadn't done it, and Xen should assert inphys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.How about we promote
VIRTUAL_BUG_ON()to expand to at leastASSERT()?
That would be good. Frankly, we discovered this because we used ASSERT in Arm and then noticed that x86 was using VIRTUAL_BUG_ON.
Quoted 4 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long
Quoted 3 lines
xen/arch/x86/numa.c +0
-0@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)
Quoted 8 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273
Quoted 4 lines
xen/arch/x86/numa.c +1
-0/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap);But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.
Yes, you're right. Actually, we just only used 1 slot. But furthermore, memnodemap[0] may be set in acpi_scan_nodes, but acpi_scan_nodes doesn't reset memnodemap when it failed. I think maybe we can add:
memnodemap[0] = 0;
memnodemapsize = 1;How do you think about it?
Quoted 1 line
Jan
RE: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
In reply to Jan BeulichHi Jan,
Quoted 20 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:37
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI inKconfigxen/arch/x86/Kconfig config X86 · line 27
Quoted 8 lines · Wei Chen
xen/arch/x86/Kconfig +1
-1@@ -24,7 +24,7 @@ config X86 select HAS_UBSAN select HAS_VPCI if HVM select NEEDS_LIBELF - select NUMA + select ACPI_NUMAWe try to keep this alphabetically sorted, so please move up the replacement line. Then
Quoted 2 lines
Reviewed-by Jan Beulich <[email protected]>
Thanks, I will do it in next version.
Quoted 1 line
Jan
RE: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
In reply to Julien GrallHi Julien,
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月13日 0:55
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI inKconfig
Quoted 6 lines
Hi Wei,
Quoted 1 line · Wei Chen
In current Xen code only implments x86 ACPI-based NUMA support.
s/implments/implements/
Got it.
Quoted 10 lines
Quoted 6 lines
So in Xen
Kconfigsystem, NUMA equals toACPI_NUMA. x86 selects NUMA by default, andCONFIG_ACPI_NUMAis hardcode inconfig.h.In this patch series, we introduced device tree based NUMA for Arm.
The concept of patch series only applies to the ML. Once checked-in this is only a series of commit. So I would write:
Quoted 7 lines
"In a follow-up patch, we will introduce support for NUMA using the Device-Tree".
Quoted 1 line
That means we will have two NUMA implemetations, so in this
s/implemetations/implementations/
Thanks, I will update the commit log in next version.
Quoted 7 lines
Quoted 2 lines
patch we decouple NUMA from ACPI based NUMA in
Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node
In reply to Wei ChenQuoted 17 lines · Wei Chen
Quoted 8 lines
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:33xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long
Quoted 3 lines · Wei Chen
xen/arch/x86/numa.c +0
-0@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)Quoted 8 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273
Quoted 4 lines
xen/arch/x86/numa.c +1
-0/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap);But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.
Quoted 6 lines
Yes, you're right. Actually, we just only used 1 slot. But furthermore, memnodemap[0] may be set in
acpi_scan_nodes, butacpi_scan_nodesdoesn't reset memnodemap when it failed. I think maybe we can add:memnodemap[0] = 0; memnodemapsize = 1;How do you think about it?
Well, yes, if data may have been put there, then resetting of course makes sense.
Jan
Re: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart
In reply to Wei ChenHi Wei,
Quoted 10 lines · Wei Chen
Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -
Quoted 4 lines
In this patch, we increased the print loops and shift bits to make putn print 64-bit number.
Signed-off-by Wei Chen <[email protected]>
Quoted 9 lines
--- xen/arch/arm/arm64/head.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)xen/arch/arm/arm64/head.S +0
-0
xen/arch/arm/arm64/head.S puts: · line 865
Quoted 5 lines
xen/arch/arm/arm64/head.S +1
-1@@ -862,17 +862,18 @@ puts: ret ENDPROC(puts) -/* Print a 32-bit number in hex. Specific to the PL011 UART. +/* Print a 64-bit number in hex. Specific to the PL011 UART.
As you modify the line, can you take the opportunity to write:
/*
* Print a 64-bit...
And also drop the second sentence as it the code has not been PL011 specific for quite a while now.
Quoted 8 lines
xen/arch/arm/arm64/head.S +1
-1* x0: Number to print. * x23: Early UART base address * Clobbers x0-x3 */ +#define PRINT_MASK 0xf000000000000000 putn: adr x1, hex - mov x3, #8
xen/arch/arm/arm64/head.S puts: · line 879
Quoted 10 lines
xen/arch/arm/arm64/head.S +3
-2+ mov x3, #16 1: early_uart_ready x23, 2 - and x2, x0, #0xf0000000 /* Mask off the top nybble */ - lsr x2, x2, #28 + and x2, x0, #PRINT_MASK /* Mask off the top nybble */ + lsr x2, x2, #60 ldrb w2, [x1, x2] /* Convert to a char */ early_uart_transmit x23, w2 lsl x0, x0, #4 /* Roll it through one nybble at a time */
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Wei ChenHi,
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,
s/resonable/reasonable/
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize() is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.
Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Quoted 40 lines
Only
platform setting can overridedma_bitsize. So in this patch, we return defaultdma_bitsize, when platform andplatorm->dma_bitsizeare not set.Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/platform.c | 4 +++- xen/common/page_alloc.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)xen/arch/arm/platform.c +3
-1@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;static bool __init platform_is_compatible (const struct platform_desc *plat) {@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)unsigned int arch_get_dma_bitsize (void ) { -return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize :32 ; +return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize + : dma_bitsize; }/* xen/common/page_alloc.c +0
-0
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 233
Quoted 8 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;integer_param ("dma_bits" , dma_bitsize);/* Offlined page list, protected by heap_lock. */
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
In reply to Wei ChenHi,
I guess this patch may be dropped after my comment on patch #4. I will comment just on the process.
Quoted 13 lines · Wei Chen
From: Hongda Deng <[email protected]>
In previous patch, we make
arch_get_dma_bitsizereturn 0 whendma_bitsizeandplatform->dma_bitsizeare not set. But this will affectlowmem_bitsizeinallocate_memory_11for domain0. Because this function dependslowmem_bitsizeto allocate memory below 4GB.In current code, when
arch_get_dma_bitsizereturn 0,lowmem_bitsizewill be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.
Quoted 4 lines
This behavior is inconsistent with what
allocate_memory_11done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before setlowmem_bitsize. Avoid settinglowmem_bitsizeto 0 by mistake.
In general, we want to avoid breaking bisection within a series. This means that this patch should be before patch #4.
Quoted 16 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/domain_build.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)xen/arch/arm/domain_build.c +0
-1@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain *d,int i;bool lowmem = true; -unsigned int lowmem_bitsize =min (32U ,arch_get_dma_bitsize ());
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 277
Quoted 10 lines
xen/arch/arm/domain_build.c +8
-0+unsigned int lowmem_bitsize =arch_get_dma_bitsize ();unsigned int bits; +/* +When dma_bitsize and platform->dma_bitsize are not set, +arch_get_dma_bitsize will return 0. That means this system +doesn't need to reserve memory for DMA. But in order to +meet above requirements, we still need to try to allocate +memory below 4GB for Dom0. +*/
The coding style for comments is:
/*
* A
* B
*/
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 282
Quoted 6 lines
xen/arch/arm/domain_build.c +2
-0+ lowmem_bitsize = lowmem_bitsize ?min (32U , lowmem_bitsize) :32U ; +/* * TODO: Implement memory bank allocation when DOM0 is not direct * mapped
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Wei ChenHi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.
What do you mean by "could not be enabled"?
Quoted 15 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0
xen/include/asm-arm/numa.h line 8
Quoted 4 lines
xen/include/asm-arm/numa.h +1
-0@@ -5,6 +5,8 @@typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )
NIT: We tend to use #ifndef rather than #if !defined(...)
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables:
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Wei ChenHi,
Quoted 14 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/arch/arm/Kconfig +0
-0
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 4 lines
xen/arch/arm/Kconfig +1
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xen is an alternative to device tree on ARM64. +config DEVICE_TREE_NUMA
The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI in the future).
xen/arch/arm/Kconfig config ACPI · line 45
Quoted 9 lines
xen/arch/arm/Kconfig +9
-0+ bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" if UNSUPPORTED + depends on ARM_64 + select NUMA + ---help--- + + Non-Uniform Memory Access (NUMA) is a computer memory design used in + multiprocessing, where the memory access time depends on the memory + location relative to the processor. +
Cheers,
Signature · 1 line
Julien Grall
Hi Wei,
Quoted 11 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.
Could you outline the long term plan for DT? Is it going to be used on production HW?
Cheers,
Signature · 1 line
Julien Grall
Hi Julien,
Quoted 16 lines · Julien Grall
Hi Wei,
Quoted 10 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
Quoted 3 lines
All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.
Could you outline the long term plan for DT? Is it going to be used on production HW?
Yes we are already and will continue to use this on production HW.
Some embedded hardware will have some usage of NUMA (as some embedded functions do require lots of computing power). We are doing experiments of that right now using those patches.
Cheers
Bertrand
Quoted 5 lines
Cheers,
Signature · 1 line
Julien Grall
Quoted 1 line · Bertrand Marquis
Hi Julien,
Hi Bertrand,
Quoted 16 lines
Quoted 16 lines · Julien Grall
Hi Wei,
Quoted 10 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
Quoted 6 lines
Quoted 3 lines
All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.
Could you outline the long term plan for DT? Is it going to be used on production HW?
Yes we are already and will continue to use this on production HW.
Some embedded hardware will have some usage of NUMA (as some embedded functions do require lots of computing power).
Interesting! Thank you for the clarifications.
Quoted 1 line
We are doing experiments of that right now using those patches.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Wei ChenHi Wei,
Quoted 9 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
Quoted 6 lines
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodeto
s/prvent/prevent/
Quoted 25 lines
parse memory nodes in EFI boot.
As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h>
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 340
Quoted 6 lines
xen/arch/arm/bootfdt.c +2
-1@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ; -if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI,
"If the system boot". Although, I would suggest to write:
"If Xen has been booted via UEFI, the memory banks will already be populated. So we should skip the parsing."
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 344
Quoted 4 lines
xen/arch/arm/bootfdt.c +4
-0+* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) )
arch_efi_enabled() is going to be less expensive than device_tree_node_matches(). So I would suggest to re-order the operands.
Quoted 46 lines
xen/arch/arm/bootfdt.c +0
-0rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem); else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )xen/arch/arm/efi/efi-boot.h +0
-25@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions. xen/include/asm-arm/setup.h +0
-0
xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32 address_cells, · line 130
Quoted 8 lines
xen/include/asm-arm/setup.h +5
-0@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32 address_cells,u32 device_tree_get_u32 (const void *fdt,int node,const char *prop_name,u32 dflt); +#if defined(CONFIG_ARM_64 ) +#define arch_efi_enabled (x) efi_enabled(x) +#else +#define arch_efi_enabled (x) (0) +#endif
I would prefer if we introduce CONFIG_EFI that would stub efi_enabled for architecture not supporting EFI.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA
In reply to Wei ChenHi Wei,
Quoted 5 lines · Wei Chen
Like
acpi_numain x86 as a switch for ACPI based NUMA, we introducedevice_tree_numaas a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.
The hypervisor will never use both ACPI and DT at runtime. In fact...
Quoted 3 lines
Keep using
bad_sratandsrat_disabledfunctions name, because we will reusenode_covers_memoryandacpi_scan_nodescode for Arm.
... given that both functions will be called from the common code, it will be a lot more difficult to add ACPI afterwards.
So I think we should either rename acpi_numa to something more generic (maybe fw_numa) or convert numa_off to a tri-state.
This will allow to have the code mostly common.
Quoted 47 lines
These
functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ 3 files changed, 38 insertions(+)create mode 100644
xen/arch/arm/numa_device_tree.cxen/arch/arm/Makefile +1
-0@@ -36,6 +36,7 @@ obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.oxen/arch/arm/numa_device_tree.c new file +16
-0@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License
xen/arch/arm/numa_device_tree.c line 26
Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +10
-0+* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +s8 device_tree_numa =0 ; + +int srat_disabled (void )
You export this one and ...
xen/arch/arm/numa_device_tree.c line 31
Quoted 1 line
xen/arch/arm/numa_device_tree.c new file +1
-0+void __init bad_srat (void )
... this one without providing in a prototype.
Looking at the rest of the series... they will be turned static in the next patch (#21) but then re-exported in patch #33.
In general, we should refrain to modify code that was added in the same patch unless it is not possible for split reason (e.g code clean-up and then code movement).
In this case, the helpers should be exported from now.
Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +4
-0+{ +printk (KERN_ERR "DT: NUMA information is not used.\n" ); + device_tree_numa =-1 ; +}xen/include/asm-arm/numa.h +0
-0
xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 30
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -23,6 +23,8 @@ typedef u8 nodeid_t;#define NUMA_LOCAL_DISTANCE 10#define NUMA_REMOTE_DISTANCE 20 +extern s8 device_tree_numa; +extern void numa_init (bool acpi_off);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenHi Wei,
Quoted 37 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)xen/arch/arm/numa_device_tree.c +5
-2@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ;
xen/arch/arm/numa_device_tree.c line 47
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +9
-0} + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ;
You seem to have a mix of soft and hard tab in this file. Is there a lot of the code that was directly copied from Linux? If not, then the file should be using Xen coding style.
xen/arch/arm/numa_device_tree.c line 59
Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12
-0+ } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node)
AFAICT, you are going to turn this helper static in a follow-up patch. This is a bad practice. Instead, the function should be static from the beginning. If it is not possible, then you should re-order the code.
In this case, I think you can add the boilerplate to parse the NUMA information (patch #25) here and then extend it in each patch.
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +3
-0+{ +uint32_t nid; +
xen/arch/arm/numa_device_tree.c line 72
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenQuoted 31 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }
xen/arch/arm/numa_device_tree.c line 41
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)
I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
Quoted 22 lines
xen/arch/arm/numa_device_tree.c +21
-0+{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; +
xen/arch/arm/numa_device_tree.c line 72
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenHi Wei,
Quoted 17 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)xen/arch/arm/numa_device_tree.c +0
-0
xen/arch/arm/numa_device_tree.c line 23
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h>
Nothing in this file seems to depend on xen/device_tree.h. So why do you need to include it?
Quoted 52 lines
xen/arch/arm/numa_device_tree.c +38
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart
In reply to Julien GrallHi Julien,
Quoted 46 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:05
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uartHi Wei,
Quoted 10 lines · Wei Chen
Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -Quoted 4 lines
In this patch, we increased the print loops and shift bits to make putn print 64-bit number.
Signed-off-by Wei Chen <[email protected]>Acked-by Julien Grall <[email protected]>Quoted 9 lines
--- xen/arch/arm/arm64/head.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)xen/arch/arm/arm64/head.S +0
-0xen/arch/arm/arm64/head.S puts: · line 865
Quoted 5 lines
xen/arch/arm/arm64/head.S +1
-1@@ -862,17 +862,18 @@ puts: ret ENDPROC(puts) -/* Print a 32-bit number in hex. Specific to the PL011 UART. +/* Print a 64-bit number in hex. Specific to the PL011 UART.As you modify the line, can you take the opportunity to write:
Quoted 6 lines
/*
* Print a 64-bit...And also drop the second sentence as it the code has not been PL011 specific for quite a while now.
Ok, I will do it in next version
Quoted 24 lines
Quoted 7 lines
xen/arch/arm/arm64/head.S +1
-1* x0: Number to print. * x23: Early UART base address * Clobbers x0-x3 */ +#define PRINT_MASK 0xf000000000000000 putn: adr x1, hex - mov x3, #8xen/arch/arm/arm64/head.S puts:
Quoted 10 lines
xen/arch/arm/arm64/head.S +3
-2+ mov x3, #16 1: early_uart_ready x23, 2 - and x2, x0, #0xf0000000 /* Mask off the top nybble */ - lsr x2, x2, #28 + and x2, x0, #PRINT_MASK /* Mask off the top nybble */ + lsr x2, x2, #60 ldrb w2, [x1, x2] /* Convert to a char */ early_uart_transmit x23, w2 lsl x0, x0, #4 /* Roll it through one nybble ata time */
Quoted 1 line
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Julien GrallHi Julien,
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setHi,
Quoted 8 lines
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 7 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.
Quoted 9 lines
Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may not have kind of restriction on hardware." ?
Quoted 5 lines
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
How about return 0 when platform doesn't specify the limit?
In my opinion, arbitrary to give 32 on AArch64 doesn't make sense. But as you mentioned, if Xen is running on a platform with DMA limitation, but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from platform->dma_bitsize. In this case, return 0 will also cause some issue.
Quoted 3 lines
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).
For Dom0, we squash the patch#5 into this patch?
Quoted 3 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Without introducing new flag, such as lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...
if ( !dma_bitsize && (num_online_nodes() > 1) )
dma_bitsize = arch_get_dma_bitsize();Quoted 58 lines
xen/arch/arm/platform.c extern const struct platform_desc _splatform[],
Quoted 17 lines
Only
platform setting can overridedma_bitsize. So in this patch, we return defaultdma_bitsize, when platform andplatorm->dma_bitsizeare not set.Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/platform.c | 4 +++- xen/common/page_alloc.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)xen/arch/arm/platform.c +0
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],_eplatform[];
xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[];
Quoted 6 lines
xen/arch/arm/platform.c +1
-0/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize; static bool __init platform_is_compatible(const struct platform_desc*plat)
xen/arch/arm/platform.c bool platform_device_is_blacklisted(const struct
Quoted 2 lines
xen/arch/arm/platform.c +0
-0{@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct
dt_device_node*node)Quoted 17 lines
xen/arch/arm/platform.c +1
-1unsigned int arch_get_dma_bitsize (void ) { -return ( platform && platform->dma_bitsize ) ? platform- dma_bitsize :32 ; + return ( platform && platform->dma_bitsize ) ? platform-
dma_bitsizexen/arch/arm/platform.c +1
-0+ : dma_bitsize; }/* xen/common/page_alloc.c +0
-0xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 233
Quoted 8 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;integer_param ("dma_bits" , dma_bitsize);/* Offlined page list, protected by heap_lock. */ Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
In reply to Julien GrallHi Julien,
Quoted 9 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:32
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 05/40] xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0
Quoted 5 lines
Hi,
I guess this patch may be dropped after my comment on patch #4. I will comment just on the process.
Ok
Quoted 50 lines · Wei Chen
Quoted 13 lines
From: Hongda Deng <[email protected]>
In previous patch, we make
arch_get_dma_bitsizereturn 0 whendma_bitsizeandplatform->dma_bitsizeare not set. But this will affectlowmem_bitsizeinallocate_memory_11for domain0. Because this function dependslowmem_bitsizeto allocate memory below 4GB.In current code, when
arch_get_dma_bitsizereturn 0,lowmem_bitsizewill be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.Quoted 4 lines
This behavior is inconsistent with what
allocate_memory_11done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before setlowmem_bitsize. Avoid settinglowmem_bitsizeto 0 by mistake.In general, we want to avoid breaking bisection within a series. This means that this patch should be before patch #4.
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain
Quoted 12 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/domain_build.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)xen/arch/arm/domain_build.c +0
-0@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain*d,
Quoted 4 lines
xen/arch/arm/domain_build.c +0
-1int i;bool lowmem = true; -unsigned int lowmem_bitsize =min (32U ,arch_get_dma_bitsize ());xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 277
Quoted 10 lines
xen/arch/arm/domain_build.c +8
-0+unsigned int lowmem_bitsize =arch_get_dma_bitsize ();unsigned int bits; +/* +When dma_bitsize and platform->dma_bitsize are not set, +arch_get_dma_bitsize will return 0. That means this system +doesn't need to reserve memory for DMA. But in order to +meet above requirements, we still need to try to allocate +memory below 4GB for Dom0. +*/
Quoted 7 lines
The coding style for comments is:
/*
* A
* B
*/
I will fix it.
Quoted 11 lines
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 282
Quoted 6 lines
xen/arch/arm/domain_build.c +2
-0+ lowmem_bitsize = lowmem_bitsize ?min (32U , lowmem_bitsize) :32U ; +/* * TODO: Implement memory bank allocation when DOM0 is not direct * mapped Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Julien GrallHi Julien,
Quoted 9 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Quoted 7 lines
Hi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA. In this case, this Kconfig option could not be enabled on Arm32.
Quoted 17 lines
Quoted 17 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0@@ -5,6 +5,8 @@
Quoted 6 lines
xen/include/asm-arm/numa.h line 8
Quoted 3 lines
xen/include/asm-arm/numa.h +1
-0typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )NIT: We tend to use #ifndef rather than #if !defined(...)
OK, I will change related changes in this series.
Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Julien GrallHi Julien,
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 1:35
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFIHi Wei,
Quoted 9 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
Quoted 9 lines
Quoted 6 lines
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodetos/prvent/prevent/
Ok
Quoted 28 lines
Quoted 28 lines
parse memory nodes in EFI boot.
As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ;
Quoted 9 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 340
Quoted 3 lines
xen/arch/arm/bootfdt.c +2
-1-if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, "If the system boot". Although, I would suggest to write:
"If Xen has been booted via UEFI, the memory banks will already be populated. So we should skip the parsing."
Yes, that would be better. I will change it in next version.
Quoted 8 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 344
Quoted 4 lines
xen/arch/arm/bootfdt.c +4
-0+* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) )
arch_efi_enabled()is going to be less expensive thandevice_tree_node_matches(). So I would suggest to re-order the operands.
yes.
Quoted 52 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 346
Quoted 2 lines
xen/arch/arm/bootfdt.c +0
-0rc = process_memory_node(fdt, node, name, depth, address_cells, size_cells,&bootinfo.mem);
xen/arch/arm/efi/efi-boot.h EFI_STATUS __init
Quoted 6 lines
xen/arch/arm/bootfdt.c +0
-0else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )xen/arch/arm/efi/efi-boot.h +0
-0@@ -197,33 +197,8 @@ EFI_STATUS __initfdt_add_uefi_nodes(EFI_SYSTEM_TABLE
*sys_table,xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32
Quoted 38 lines
xen/arch/arm/efi/efi-boot.h +0
-25int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions. xen/include/asm-arm/setup.h +0
-0@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32
address_cells,Quoted 3 lines
xen/include/asm-arm/setup.h +0
-0u32 device_tree_get_u32 (const void *fdt,int node,const char *prop_name,u32 dflt);
Quoted 9 lines
xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32 address_cells, · line 130
Quoted 5 lines
xen/include/asm-arm/setup.h +5
-0+#if defined(CONFIG_ARM_64 ) +#define arch_efi_enabled (x) efi_enabled(x) +#else +#define arch_efi_enabled (x) (0) +#endif I would prefer if we introduce
CONFIG_EFIthat would stubefi_enabledfor architecture not supporting EFI.
Yes, that's a good idea.
I also feel a little awkward with the current arch helpers.
Quoted 4 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA
In reply to Julien GrallHi Julien,
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 1:45
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 21/40] xen/arm: introducedevice_tree_numaas a switch for device tree NUMAHi Wei,
Quoted 9 lines
Quoted 5 lines · Wei Chen
Like
acpi_numain x86 as a switch for ACPI based NUMA, we introducedevice_tree_numaas a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.The hypervisor will never use both ACPI and DT at runtime. In fact...
Yes.
Quoted 4 lines
Quoted 3 lines
Keep using
bad_sratandsrat_disabledfunctions name, because we will reusenode_covers_memoryandacpi_scan_nodescode for Arm.
Quoted 8 lines
... given that both functions will be called from the common code, it will be a lot more difficult to add ACPI afterwards.
So I think we should either rename
acpi_numato something more generic (maybefw_numa) or convertnuma_offto a tri-state.This will allow to have the code mostly common.
I will try to address them in next version.
Quoted 65 lines
Review
Quoted 26 lines
These
functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ 3 files changed, 38 insertions(+)create mode 100644
xen/arch/arm/numa_device_tree.cxen/arch/arm/Makefile +1
-0@@ -36,6 +36,7 @@ obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.o(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 21 lines
new file mode 100644
index 0000000000..1c74ad135dxen/arch/arm/numa_device_tree.c new file +16
-0@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License xen/arch/arm/numa_device_tree.c line 26
Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +10
-0+* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +s8 device_tree_numa =0 ; + +int srat_disabled (void )You export this one and ...
xen/arch/arm/numa_device_tree.c line 31
Quoted 1 line
xen/arch/arm/numa_device_tree.c new file +1
-0+void __init bad_srat (void )... this one without providing in a prototype.
Quoted 9 lines
Looking at the rest of the series... they will be turned static in the next patch (#21) but then re-exported in patch #33.
In general, we should refrain to modify code that was added in the same patch unless it is not possible for split reason (e.g code clean-up and then code movement).
In this case, the helpers should be exported from now.
Ok.
Quoted 24 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c new file +4
-0+{ +printk (KERN_ERR "DT: NUMA information is not used.\n" ); + device_tree_numa =-1 ; +}xen/include/asm-arm/numa.h +0
-0xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 29
Quoted 7 lines
xen/include/asm-arm/numa.h +2
-0@@ -23,6 +23,8 @@ typedef u8 nodeid_t;#define NUMA_LOCAL_DISTANCE 10#define NUMA_REMOTE_DISTANCE 20 +extern s8 device_tree_numa; +extern void numa_init (bool acpi_off); extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_tdistance);
Quoted 2 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Julien GrallHi Julien,
Quoted 26 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.c
Quoted 12 lines
xen/arch/arm/numa_device_tree.c line 23
Quoted 8 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> Nothing in this file seems to depend on
xen/device_tree.h. So why do you need to include it?
I remember that without this header file, device_tree_get_u32 in this patch will cause compiling failed.
Quoted 56 lines
Quoted 36 lines
xen/arch/arm/numa_device_tree.c +23
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} +xen/arch/arm/numa_device_tree.c
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}--
Julien Grall
RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Julien GrallHi Julien,
Quoted 35 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 14 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/arch/arm/Kconfig +0
-0xen/arch/arm/Kconfig config ACPI · line 33
Quoted 1 line
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xenis
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAThe name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
Quoted 7 lines
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI in the future).
How about we select DT_NUMA default on Arm64. And DT_NUMA select NUMA like what we have done in patch#6 in x86? And remove the description?
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
Quoted 17 lines
xen/arch/arm/Kconfig config ACPI · line 37
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" ifUNSUPPORTED
xen/arch/arm/Kconfig config ACPI
Quoted 5 lines
xen/arch/arm/Kconfig +5
-0+ depends on ARM_64 + select NUMA + ---help--- + + Non-Uniform Memory Access (NUMA) is a computer memory design usedin
xen/arch/arm/Kconfig config ACPI · line 43
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ multiprocessing, where the memory access time depends on thememory
xen/arch/arm/Kconfig config ACPI · line 45
Quoted 2 lines
xen/arch/arm/Kconfig +2
-0+ location relative to the processor. +Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 35 lines
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setHi,
Quoted 8 lines
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 13 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.
Quoted 7 lines
Quoted 3 lines
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may not have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
Quoted 10 lines
Quoted 7 lines
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
How about return 0 when platform doesn't specify the limit? In my opinion, arbitrary to give 32 on AArch64 doesn't make sense.
We have to care about the common use-case. We added this restriction a few years ago because we had a few reports of users using only 32-bit DMA capable device.
Quoted 10 lines
But as you mentioned, if Xen is running on a platform with DMA limitation, but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from
platform->dma_bitsize. In this case, return 0 will also cause some issue.Quoted 3 lines
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).
For Dom0, we squash the patch#5 into this patch?
Let me answer with another question. Why should we modify the Arm code rather than the common code? IOW...
Quoted 10 lines
Quoted 4 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();
... Why do you think we need this check on Arm when NUMA is enabled?
We can discuss how to remove it once this is answered.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 12 lines
Quoted 12 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIHi Wei,
Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.
... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0@@ -5,6 +5,8 @@
Quoted 28 lines
Quoted 6 lines
xen/include/asm-arm/numa.h line 8
Quoted 3 lines
xen/include/asm-arm/numa.h +1
-0typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )NIT: We tend to use #ifndef rather than #if !defined(...)
OK, I will change related changes in this series.
Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: Cheers,
--
Julien Grall
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 39 lines
Quoted 39 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 14 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/arch/arm/Kconfig +0
-0xen/arch/arm/Kconfig config ACPI · line 33
Quoted 1 line
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xenis
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAThe name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
Quoted 6 lines
Quoted 3 lines
So we should have a generic config that will then select DT (and ACPI in the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?
I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 39 lines
Quoted 26 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 12 lines
xen/arch/arm/numa_device_tree.c line 23
Quoted 8 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> Nothing in this file seems to depend on
xen/device_tree.h. So why do you need to include it?
Quoted 2 lines
I remember that without this header file,
device_tree_get_u32in this patch will cause compiling failed.
I looked at the prototype of device_tree_get_u32() and I can't find how it depends on bits from device_tree.h. Can you paste the compilation error?
Quoted 48 lines
Quoted 48 lines
Quoted 37 lines
xen/arch/arm/numa_device_tree.c +23
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} +xen/arch/arm/numa_device_tree.c
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
Quoted 9 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}--
Julien Grall
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Julien GrallHi Julien,
Quoted 59 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 7 lines
Quoted 7 lines
when platform is not set
Hi,
Quoted 2 lines · Wei Chen
From: Hongda Deng <[email protected]>
Quoted 14 lines
Quoted 5 lines
Quoted 2 lines
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 6 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by everydevices.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.Quoted 6 lines
Quoted 3 lines
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may
Quoted 8 lines
not
Quoted 1 line
have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
OK, I understand now. Let's keep 32-bit as default value, but even in this case, how about DMA-16 devices? Although these devices are very rare, they still exist : )
Anyway, keep the arch_get_dma_bitsize as original maybe the best approach for now.
Quoted 22 lines
Quoted 10 lines
Quoted 7 lines
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
How about return 0 when platform doesn't specify the limit? In my opinion, arbitrary to give 32 on AArch64 doesn't make sense.
We have to care about the common use-case. We added this restriction a few years ago because we had a few reports of users using only 32-bit DMA capable device.
Quoted 1 line
But as you mentioned, if Xen is running on a platform with DMA
limitation,
Quoted 6 lines
but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from
platform->dma_bitsize. In this case, return 0 will also cause some issue.Quoted 2 lines
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for
Quoted 7 lines
now).
Quoted 3 lines
Quoted 1 line
For Dom0, we squash the patch#5 into this patch?
Let me answer with another question. Why should we modify the Arm code rather than the common code? IOW...
As I answered above, let's keep arch_get_dma_bitsize as original then we don't need to modify Arm code.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Quoted 8 lines
Quoted 5 lines
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();... Why do you think we need this check on Arm when NUMA is enabled?
I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this check.
Quoted 2 lines
We can discuss how to remove it once this is answered.
I think we can start to discuss it.
Quoted 4 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Julien GrallHi Julien,
Quoted 41 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 4 lines
Quoted 4 lines
NUMA API
Hi Wei,
Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...
Quoted 6 lines
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.
Ok, I understand your concern. Yes, my words would lead to mis-understanding. If we make CONFIG_NUMA enabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed. I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use? And how can we test this code? Before this patch, I had checked Linux, and found that OF_NUMA only selected by Arm64 not Arm32. But if you feel the need to add to arm32, I have no problem with that.
Quoted 51 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0@@ -5,6 +5,8 @@Quoted 28 lines
Quoted 6 lines
xen/include/asm-arm/numa.h line 8
Quoted 3 lines
xen/include/asm-arm/numa.h +1
-0typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )NIT: We tend to use #ifndef rather than #if !defined(...)
OK, I will change related changes in this series.
Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: Cheers,
--
Julien GrallCheers,
--
Julien Grall
RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Julien GrallHi Julien,
Quoted 56 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 21 lines
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 9 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig config ACPI · line 33
Quoted 6 lines
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) supportfor Xen
Quoted 5 lines
Quoted 5 lines
is
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAQuoted 8 lines
Quoted 8 lines
The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI
in
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Ok. I agree with this.
Quoted 8 lines
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.
Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already have CONFIG_NUMA in common layer. So we need to add another one for Arm like CONFIG_ARM_NUMA?
And in this option, we can select CONFIG_DEVICE_TREE_NUMA automatically if device tree is enabled. If CONFIG_ACPI is enabled, we will select CONFIG_ACPI_NUMA too (in the future)
In Xen code, DT_NUMA and ACPI_NUMA code can co-exist, Xen will check the system ACPI support status to decide to use DT_NUMA or ACPI_NUMA?
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 68 lines
Quoted 59 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 7 lines
Quoted 7 lines
when platform is not set
Hi,
Quoted 2 lines · Wei Chen
From: Hongda Deng <[email protected]>
Quoted 14 lines
Quoted 5 lines
Quoted 2 lines
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 6 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by everydevices.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.Quoted 6 lines
Quoted 3 lines
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They mayQuoted 8 lines
not
Quoted 1 line
have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
Quoted 3 lines
OK, I understand now. Let's keep 32-bit as default value, but even in this case, how about DMA-16 devices? Although these devices are very rare, they still exist : )
I haven't heard anyone reporting issues with them on Xen on Arm. So I assume that either it works or no-one is using them.
My main point is we need to care about the common use case. 32-bit DMA device is still a thing and caused trouble to some of our users (e.g. NXP).
If tomorrow, someone report issue with 16-bit DMA device, then we can consider our options how to handle.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Quoted 8 lines
Quoted 5 lines
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();... Why do you think we need this check on Arm when NUMA is enabled?
Quoted 7 lines
I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this check.
Quoted 2 lines
We can discuss how to remove it once this is answered.
I think we can start to discuss it.
How about replacing the second part of the check with a new helper arch_have_default_dma_zone() (or a different name)?
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 45 lines
Quoted 45 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 4 lines
Quoted 4 lines
NUMA API
Hi Wei,
Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.
Quoted 7 lines
Quoted 3 lines
... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.
Ok, I understand your concern. Yes, my words would lead to mis-understanding. If we make
CONFIG_NUMAenabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.
When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?
Quoted 11 lines
I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?
Technically you already added the code because arch/arm/ is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...
And how can we test this code?
I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 78 lines
Quoted 57 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 21 lines
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 9 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig config ACPI · line 33
Quoted 6 lines
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) supportfor Xen
Quoted 5 lines
Quoted 5 lines
is
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAQuoted 8 lines
Quoted 8 lines
The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI
in
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Ok. I agree with this.
Quoted 8 lines
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
Quoted 7 lines
No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.
Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already haveCONFIG_NUMAin common layer. So we need to add another one for Arm likeCONFIG_ARM_NUMA?
I think so.
Quoted 5 lines
And in this option, we can select
CONFIG_DEVICE_TREE_NUMAautomatically if device tree is enabled. IfCONFIG_ACPIis enabled, we will selectCONFIG_ACPI_NUMAtoo (in the future)
In Xen code,DT_NUMAandACPI_NUMAcode can co-exist, Xen
Distributions should not have to build a different Xen for DT and ACPI. So it is more they *must* co-exist.
Quoted 2 lines
will check the system ACPI support status to decide to use
DT_NUMAorACPI_NUMA?
Yes. A user should only have to say "I want to use NUMA". This is Xen to figure out whether we need to compile the support for DT and/or ACPI.
Once we have support for APCI, it doesn't make a lot of sense for the users to say "I want to compile with DT and ACPI but I only want NUMA when using DT".
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Julien GrallHi Julien,
Quoted 57 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:44
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 19 lines
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41+++++++++++++++++++++++++++++++--
Quoted 4 lines
Quoted 4 lines
Review
Quoted 3 lines
1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 10 lines
Quoted 10 lines
xen/arch/arm/numa_device_tree.c line 23
Quoted 8 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> Nothing in this file seems to depend on
xen/device_tree.h. So why doyou
Quoted 3 lines
Quoted 2 lines
need to include it?
Quoted 7 lines
Quoted 1 line
I remember that without this header file,
device_tree_get_u32in thispatch
Quoted 1 line
will cause compiling failed.
I looked at the prototype of
device_tree_get_u32()and I can't find how it depends on bits fromdevice_tree.h. Can you paste the compilation error?
I tested it again, this header file should be introduced in following patches:numa_device_tree.c: In function ‘device_tree_parse_numa_distance_map_v1’:
numa_device_tree.c:243:16: error: implicit declaration of function ‘dt_read_number’ [-Werror=implicit-function-declaration]
243 | from = dt_read_number(matrix, 1);
| ^~~~~~~~~~~~~~numa_device_tree.c:243:16: error: nested extern declaration of ‘dt_read_number’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
I will move it to another patch.
Quoted 63 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
xen/arch/arm/numa_device_tree.c +21
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ;Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid = device_tree_get_u32(fdt, node, "numa-node-id",
MAX_NUMNODES);Quoted 5 lines
Quoted 5 lines
xen/arch/arm/numa_device_tree.c
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
Quoted 9 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}--
Julien GrallCheers,
--
Julien Grall
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Julien GrallHi Julien,
Quoted 120 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:18
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 30 lines
Quoted 30 lines
when platform is not set
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 13 lines
Quoted 12 lines
when platform is not set
Hi,
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Ok
Quoted 5 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We needto
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
be able to allocate some DMA-able memory that can be used by every
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against ourhardware
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
design. We don't have that kind of restriction on hardware.
What do you mean by "hardware design"? Are you referring to the
server
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They maynot
Quoted 1 line
have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged).
What
Quoted 5 lines
Quoted 3 lines
you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
OK, I understand now. Let's keep 32-bit as default value, but even in
this
Quoted 1 line
case, how about DMA-16 devices? Although these devices are very rare,
they
Quoted 1 line
still exist : )
I haven't heard anyone reporting issues with them on Xen on Arm. So I assume that either it works or no-one is using them.
My main point is we need to care about the common use case. 32-bit DMA device is still a thing and caused trouble to some of our users (e.g. NXP).
If tomorrow, someone report issue with 16-bit DMA device, then we can consider our options how to handle.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I would explore to remove the NUMA check for drop the DMA zone.
FAOD,
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
both suggestion are for Arm only. For x86, they need to be kept.
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();Quoted 5 lines
Quoted 2 lines
... Why do you think we need this check on Arm when NUMA is enabled?
I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this
check.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
We can discuss how to remove it once this is answered.
I think we can start to discuss it.
How about replacing the second part of the check with a new helper
arch_have_default_dma_zone()(or a different name)?
This seems like a method worth trying, I will try to follow this way to compose next version.
Quoted 5 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Julien GrallHi Julien,
Quoted 68 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 26 lines
Quoted 26 lines
NUMA API
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 8 lines
Quoted 8 lines
NUMA API
Hi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And
that's
Quoted 6 lines
Quoted 6 lines
fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.... you continue to say "couldn't be enabled" without clarifying
whether
Quoted 3 lines
Quoted 2 lines
this mean that the build will break or this was just not tested because you don't have any platform.
Quoted 9 lines
Quoted 1 line
Ok, I understand your concern. Yes, my words would lead to mis-
understanding.
Quoted 2 lines
If we make
CONFIG_NUMAenabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?
It could not build because I have not tried to enable device_tree_numa option for Arm32 but enabled NUMA for arm32.
I have tested it again, yes, simple enable device_tree_numa and NUMA for arm32 can build a image successfully.
So, I think it's OK to enable this on Arm32, and I will do it in next version. But, can we still keep these FAKE APIs? If user don't want to enable NUMA they still can make Xen work? And I will remove "could not enable for Arm32" from commit log.
Quoted 12 lines
Quoted 11 lines
I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?
Quoted 8 lines
Technically you already added the code because
arch/arm/is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...And how can we test this code?
I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.
I mean, we don't have Arm32 NUMA machine to test, I don't know the code works well on Arm32 NUMA or not. I only can verify them on non-NUMA arm32, and make sure this code will not break existed machines.
Quoted 5 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Julien GrallHi Julien,
Quoted 118 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:29
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 15 lines
Quoted 15 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce
DEVICE_TREE_NUMAQuoted 24 lines
Quoted 24 lines
Quoted 14 lines
Quoted 14 lines
Kconfigfor arm64Hi,
Quoted 9 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig config ACPI · line 33
Quoted 6 lines
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) supportfor Xen
Quoted 3 lines
Quoted 3 lines
is
Quoted 2 lines
xen/arch/arm/Kconfig +0
-0an alternative to device tree on ARM64.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+config DEVICE_TREE_NUMAThe name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the
choice
Quoted 25 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI
in
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Ok. I agree with this.
Quoted 8 lines
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
Quoted 7 lines
No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.
Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already haveCONFIG_NUMAin common layer. So we need to add another one for Arm likeCONFIG_ARM_NUMA?I think so.
Quoted 5 lines
And in this option, we can select
CONFIG_DEVICE_TREE_NUMAautomatically if device tree is enabled. IfCONFIG_ACPIis enabled, we will selectCONFIG_ACPI_NUMAtoo (in the future)
In Xen code,DT_NUMAandACPI_NUMAcode can co-exist, XenDistributions should not have to build a different Xen for DT and ACPI. So it is more they *must* co-exist.
Quoted 2 lines
will check the system ACPI support status to decide to use
DT_NUMAorACPI_NUMA?
Quoted 7 lines
Yes. A user should only have to say "I want to use NUMA". This is Xen to figure out whether we need to compile the support for DT and/or ACPI.
Once we have support for APCI, it doesn't make a lot of sense for the users to say "I want to compile with DT and ACPI but I only want NUMA when using DT".
I am glad we are now in the same page. Ok, I will change the Kconfig like this in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 81 lines
Quoted 68 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 26 lines
Quoted 26 lines
NUMA API
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 8 lines
Quoted 8 lines
NUMA API
Hi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And
that's
Quoted 6 lines
Quoted 6 lines
fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.... you continue to say "couldn't be enabled" without clarifying
whether
Quoted 3 lines
Quoted 2 lines
this mean that the build will break or this was just not tested because you don't have any platform.
Quoted 9 lines
Quoted 1 line
Ok, I understand your concern. Yes, my words would lead to mis-
understanding.
Quoted 2 lines
If we make
CONFIG_NUMAenabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?
It could not build because I have not tried to enable
device_tree_numaoption for Arm32 but enabled NUMA for arm32.
Quoted 6 lines
I have tested it again, yes, simple enable
device_tree_numaand NUMA for arm32 can build a image successfully.So, I think it's OK to enable this on Arm32, and I will do it in next version. But, can we still keep these FAKE APIs? If user don't want to enable NUMA they still can make Xen work?
Yes, we still need to keep the FAKE APIs. I was only commenting about the wording in the commit message.
Quoted 21 lines
And I will remove "could not
enable for Arm32" from commit log.Quoted 18 lines
Quoted 11 lines
I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?
Technically you already added the code because
arch/arm/is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...And how can we test this code?
Quoted 7 lines
Quoted 2 lines
I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.
I mean, we don't have Arm32 NUMA machine to test, I don't know the code works well on Arm32 NUMA or not. I only can verify them on non-NUMA arm32, and make sure this code will not break existed machines.
I understood you don't have any Arm32 NUMA machine. But I don't see the lack of testing as an issue because the code doesn't look to contain bits that may rely on arm64. So there are very limited reasons for the code to break on arm32.
If we really want to test it, then it should be feasible to fake the NUMA node in the DT. However, I don't expect you to do it.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Julien GrallHi Julien,
Quoted 56 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:10
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 30 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +10
-2@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () )
Quoted 9 lines
xen/arch/arm/numa_device_tree.c line 47
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+return -EINVAL ; + else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ;You seem to have a mix of soft and hard tab in this file. Is there a lot of the code that was directly copied from Linux? If not, then the file should be using Xen coding style.
I copied some code from x86, and x86 is Linux style. So, yes, I should adjust them it Xen coding style. I will do it in next version.
Quoted 17 lines
xen/arch/arm/numa_device_tree.c line 59
Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12
-0+ } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node)AFAICT, you are going to turn this helper static in a follow-up patch. This is a bad practice. Instead, the function should be static from the beginning. If it is not possible, then you should re-order the code.
Quoted 4 lines
In this case, I think you can add the boilerplate to parse the NUMA information (patch #25) here and then extend it in each patch.
That's make sense, I will try to address it in next version.
Quoted 20 lines
xen/arch/arm/numa_device_tree.c
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Julien GrallHi Julien,
Quoted 43 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 19 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }
Quoted 14 lines
xen/arch/arm/numa_device_tree.c line 41
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
yes, in the very beginning, I want to keep device_tree_ prefix for functions that will handle dtb file. And use dtb_ prefix to replace acpi, to indicate, this function is device tree version numa implementation.
If that's not the right reason, I will unify all prefix to device_tree_ in next version. How do you think about it?
Quoted 38 lines
Quoted 16 lines
xen/arch/arm/numa_device_tree.c +16
-0+{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} +xen/arch/arm/numa_device_tree.c
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 58 lines
Quoted 43 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 19 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }Quoted 14 lines
xen/arch/arm/numa_device_tree.c line 41
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
Quoted 3 lines
yes, in the very beginning, I want to keep
device_tree_prefix for functions that will handle dtb file. And usedtb_prefix to replace acpi, to indicate, this function is device tree version numa implementation.
Thanks for the clarification. The difference between "dtb" and "device_tree" is quite subttle: the former refers to the binary while the latter refers to the format. Most of the readers are likely to infer they mean the same. So I think this will bring more confusion.
Quoted 3 lines
If that's not the right reason, I will unify all prefix to
device_tree_in next version. How do you think about it?
AFAICT, your parsing functions will always start with "device_tree_parse_". I would prefer if the set replacing the ACPI helpers start with "device_tree_".
If you are concern with the length of the function name, then I would suggest to prefix all the functions with "fdt" (We are dealing with the flattened DT after all) or "dt".
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common
In reply to Wei ChenHi Wei,
Quoted 4 lines · Wei Chen
In the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from
arch/x86to common.
I would add "No functional changes intended" to make clear this patch is only moving code.
Quoted 148 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 114 -------------------------------- xen/common/Makefile | 1 + xen/common/numa.c | 131 +++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 29 -------- xen/include/xen/numa.h | 35 ++++++++++ 5 files changed, 167 insertions(+), 143 deletions(-)create mode 100644
xen/common/numa.cxen/arch/x86/numa.c +0
-114@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -struct node_data node_data[MAX_NUMNODES ]; - -/* Mapping from pdx to node id */ -int memnode_shift; -static typeof(*memnodemap) _memnodemap[64]; -unsigned long memnodemapsize; -u8 *memnodemap; -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { [0 ...NR_CPUS -1 ] =NUMA_NO_NODE };@@ -58,112 +50,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* -* Given a shift value, try to populate memnodemap[] -* Returns : -* 1 if OK -* 0 if memnodmap[] too small (of shift too small) -* -1 if node overlap or lost ram (shift too big) -*/ -static int __init populate_memnodemap (const struct node *nodes, -int numnodes,int shift,nodeid_t *nodeids) -{ -unsigned long spdx, epdx; -int i, res =-1 ; - -memset (memnodemap,NUMA_NO_NODE , memnodemapsize *sizeof (*memnodemap)); -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; -if ( (epdx >> shift) >= memnodemapsize ) -return 0 ; -do { -if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) -return -1 ; - -if ( !nodeids ) - memnodemap[spdx >> shift] = i; -else - memnodemap[spdx >> shift] = nodeids[i]; - - spdx += (1UL << shift); - }while ( spdx < epdx ); - res =1 ; - } - -return res; -} - -static int __init allocate_cachealigned_memnodemap (void ) -{ -unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); -unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); - - memnodemap =mfn_to_virt (mfn); - mfn <<=PAGE_SHIFT ; - size <<=PAGE_SHIFT ; -printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , - mfn, mfn + size); - memnodemapsize = size /sizeof (*memnodemap); - -return 0 ; -} - -/* -* The LSB of all start and end addresses in the node map is the value of the -* maximum possible shift. -*/ -static int __init extract_lsb_from_nodes (const struct node *nodes, -int numnodes) -{ -int i, nodes_used =0 ; -unsigned long spdx, epdx; -unsigned long bitfield =0 , memtop =0 ; - -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; - bitfield |= spdx; - nodes_used++; -if ( epdx > memtop ) - memtop = epdx; - } -if ( nodes_used <=1 ) - i =BITS_PER_LONG -1 ; -else - i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); - memnodemapsize = (memtop >> i) +1 ; -return i; -} - -int __init compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids) -{ -int shift; - - shift =extract_lsb_from_nodes (nodes, numnodes); -if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) - memnodemap = _memnodemap; -else if (allocate_cachealigned_memnodemap () ) -return -1 ; -printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); - -if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) - { -printk (KERN_INFO "Your memory is not aligned you need to " -"rebuild your hypervisor with a bigger NODEMAPSIZE " -"shift=%d\n" , shift); -return -1 ; - } - -return shift; -}/* initialize NODE_DATA given nodeid and start/end */ void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) {xen/common/Makefile +0
-0
xen/common/Makefile obj-y += wait.o · line 57
Quoted 4 lines
xen/common/Makefile +1
-0@@ -54,6 +54,7 @@ obj-y += wait.o obj-bin-y += warning.init.o obj-$(CONFIG_XENOPROF) += xenoprof.o obj-y += xmalloc_tlsf.o +obj-$(CONFIG_NUMA) += numa.o
AFAICT, the Makefile is listing the file in alphabetical order. So please add numa.o in the correct position.
Quoted 9 lines
xen/common/Makefile +0
-0obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma lzo unlzo unlz4 unzstd earlycpio,$(n).init.o)xen/common/numa.c new file +0
-0
xen/common/numa.c line 10
Quoted 10 lines
xen/common/numa.c new file +10
-0@@ -0,0 +1,131 @@ +/* +* Generic VM initialization for x86-64 NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/mm.h> +#include <xen/string.h> +#include <xen/init.h> +#include <xen/ctype.h>
You don't seem to use any helpers./types directly defined by at least this header...
xen/common/numa.c line 13
Quoted 3 lines
xen/common/numa.c new file +3
-0+#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/time.h>
... this one and ...
xen/common/numa.c line 14
Quoted 1 line
xen/common/numa.c new file +1
-0+#include <xen/smp.h>
... this one. Can you check the list of headers and introduce the minimum? If the dependency is required by another headers, then I think that dependency should be moved in the header requiring it.
xen/common/numa.c line 16
Quoted 2 lines
xen/common/numa.c new file +2
-0+#include <xen/pfn.h> +#include <xen/sched.h>
Please sort the includes in alphabetical order.
Quoted 167 lines
xen/common/numa.c new file +115
-0+ +struct node_data node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +int memnode_shift; +typeof(*memnodemap) _memnodemap[64]; +unsigned long memnodemapsize; +u8 *memnodemap; + +/* +* Given a shift value, try to populate memnodemap[] +* Returns : +* 1 if OK +* 0 if memnodmap[] too small (of shift too small) +* -1 if node overlap or lost ram (shift too big) +*/ +static int __init populate_memnodemap (const struct node *nodes, +int numnodes,int shift,nodeid_t *nodeids) +{ +unsigned long spdx, epdx; +int i, res =-1 ; + +memset (memnodemap,NUMA_NO_NODE , memnodemapsize *sizeof (*memnodemap)); +for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx ) +continue ; +if ( (epdx >> shift) >= memnodemapsize ) +return 0 ; +do { +if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) +return -1 ; + +if ( !nodeids ) + memnodemap[spdx >> shift] = i; +else + memnodemap[spdx >> shift] = nodeids[i]; + + spdx += (1UL << shift); + }while ( spdx < epdx ); + res =1 ; + } + +return res; +} + +static int __init allocate_cachealigned_memnodemap (void ) +{ +unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); +unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); + + memnodemap =mfn_to_virt (mfn); + mfn <<=PAGE_SHIFT ; + size <<=PAGE_SHIFT ; +printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , + mfn, mfn + size); + memnodemapsize = size /sizeof (*memnodemap); + +return 0 ; +} + +/* +* The LSB of all start and end addresses in the node map is the value of the +* maximum possible shift. +*/ +static int __init extract_lsb_from_nodes (const struct node *nodes, +int numnodes) +{ +int i, nodes_used =0 ; +unsigned long spdx, epdx; +unsigned long bitfield =0 , memtop =0 ; + +for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx ) +continue ; + bitfield |= spdx; + nodes_used++; +if ( epdx > memtop ) + memtop = epdx; + } +if ( nodes_used <=1 ) + i =BITS_PER_LONG -1 ; +else + i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); + memnodemapsize = (memtop >> i) +1 ; +return i; +} + +int __init compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids) +{ +int shift; + + shift =extract_lsb_from_nodes (nodes, numnodes); +if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) + memnodemap = _memnodemap; +else if (allocate_cachealigned_memnodemap () ) +return -1 ; +printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); + +if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) + { +printk (KERN_INFO "Your memory is not aligned you need to " +"rebuild your hypervisor with a bigger NODEMAPSIZE " +"shift=%d\n" , shift); +return -1 ; + } + +return shift; +}xen/include/asm-x86/numa.h +0
-29@@ -26,7 +26,6 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -#define VIRTUAL_BUG_ON (x)extern void numa_add_cpu (int cpu);extern void numa_init_array (void );@@ -47,34 +46,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -/* Simple perfect hash to map pdx to node numbers */ -extern int memnode_shift; -extern unsigned long memnodemapsize; -extern u8 *memnodemap; - -struct node_data { -unsigned long node_start_pfn; -unsigned long node_spanned_pages; -}; - -extern struct node_data node_data[]; - -static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) -{ -nodeid_t nid; -VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); - nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; -VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); -return nid; -} - -#define NODE_DATA (nid) (&(node_data[nid])) - -#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) -#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) -#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ - NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node);void srat_parse_regions (u64 addr);xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h line 22
Quoted 5 lines
xen/include/xen/numa.h +2
-0@@ -18,4 +18,39 @@ (((d)->vcpu !=NULL && (d)->vcpu[0 ] !=NULL ) \ ?vcpu_to_node ((d)->vcpu[0 ]) :NUMA_NO_NODE ) +/* The following content can be used when NUMA feature is enabled */ +#if defined(CONFIG_NUMA )
Please use #ifdef CONFIG_NUMA
xen/include/xen/numa.h line 28
Quoted 6 lines
xen/include/xen/numa.h +6
-0+ +/* Simple perfect hash to map pdx to node numbers */ +extern int memnode_shift; +extern unsigned long memnodemapsize; +extern u8 *memnodemap; +extern typeof(*memnodemap) _memnodemap[64];
AFAICT, this will be turned static against in a follow-up patch. Can this be avoided?
Quoted 19 lines
xen/include/xen/numa.h +18
-0+ +struct node_data { +unsigned long node_start_pfn; +unsigned long node_spanned_pages; +}; + +extern struct node_data node_data[]; +#define VIRTUAL_BUG_ON (x) + +static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) +{ +nodeid_t nid; +VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); + nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; +VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); +return nid; +} +
xen/include/xen/numa.h line 56
Quoted 10 lines
xen/include/xen/numa.h +9
-0+#define NODE_DATA (nid) (&(node_data[nid])) + +#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) +#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) +#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ + NODE_DATA(nid)->node_spanned_pages) + +#endif /* CONFIG_NUMA */ +#endif /* _XEN_NUMA_H */
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node to common
In reply to Wei ChenHi Wei,
Quoted 6 lines · Wei Chen
This function will be reused by Arm later, so we move it from
arch/x86to common. But we keepcpu_to_nodeandnode_to_cpumaskto x86 header file. Becausecpu_to_nodeandnode_to_cpumaskhave different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.
AFAICT, the Arm helpers are gated by !CONFIG_NUMA and the ones in common code will be gated by CONFIG_NUMA. So I am not quite too understand why they can't be moved now. Can you clarify it?
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 10/40] xen/x86: Move NR_NODE_MEMBLKS macro to common
In reply to Wei ChenHi Wei,
Quoted 2 lines · Wei Chen
Not only x86 ACPI need this macro. Device tree based NUMA also needs this macro to present max memory block number.
AFAICT, a memory range described in DT cannot be split across multiple nodes. So I think we want to define NR_NODE_MEMBLKS as NR_MEM_BANKS.
Quoted 27 lines
So we move it from x86 ACPI header file to common NUMA header file.
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-x86/acpi.h | 1 - xen/include/xen/numa.h | 1 + 2 files changed, 1 insertion(+), 1 deletion(-)xen/include/asm-x86/acpi.h +0
-1@@ -103,7 +103,6 @@ extern unsigned long acpi_wakeup_address;extern s8 acpi_numa;extern int acpi_scan_nodes (u64 start,u64 end); -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h line 17
Quoted 7 lines
xen/include/xen/numa.h +1
-0@@ -11,6 +11,7 @@#define NUMA_NO_DISTANCE 0xFF#define MAX_NUMNODES (1 << NODES_SHIFT) +#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)#define vcpu_to_node (v) (cpu_to_node((v)->processor))
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common
In reply to Julien GrallHi Julien,
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月24日 1:47
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to commonHi Wei,
Quoted 8 lines
Quoted 4 lines · Wei Chen
In the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from
arch/x86to common.I would add "No functional changes intended" to make clear this patch is only moving code.
Ok, I will do it.
Quoted 151 lines
Quoted 34 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 114 -------------------------------- xen/common/Makefile | 1 + xen/common/numa.c | 131 +++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 29 -------- xen/include/xen/numa.h | 35 ++++++++++ 5 files changed, 167 insertions(+), 143 deletions(-)create mode 100644
xen/common/numa.cxen/arch/x86/numa.c +0
-8@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -struct node_data node_data[MAX_NUMNODES ]; - -/* Mapping from pdx to node id */ -int memnode_shift; -static typeof(*memnodemap) _memnodemap[64]; -unsigned long memnodemapsize; -u8 *memnodemap; -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { [0 ...NR_CPUS -1 ] =NUMA_NO_NODE };@@ -58,112 +50,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; }xen/arch/x86/numa.c int srat_disabled(void) · line 69
Quoted 10 lines
xen/arch/x86/numa.c +0
-9-/* -* Given a shift value, try to populate memnodemap[] -* Returns : -* 1 if OK -* 0 if memnodmap[] too small (of shift too small) -* -1 if node overlap or lost ram (shift too big) -*/ -static int __init populate_memnodemap (const struct node *nodes, - int numnodes, int shift, nodeid_t*nodeids)
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 5 lines
xen/arch/x86/numa.c +0
-5-{ -unsigned long spdx, epdx; -int i, res =-1 ; - - memset(memnodemap, NUMA_NO_NODE, memnodemapsize *
sizeof(*memnodemap));Quoted 32 lines
xen/arch/x86/numa.c +0
-32-for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; -if ( (epdx >> shift) >= memnodemapsize ) -return 0 ; -do { -if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) -return -1 ; - -if ( !nodeids ) - memnodemap[spdx >> shift] = i; -else - memnodemap[spdx >> shift] = nodeids[i]; - - spdx += (1UL << shift); - }while ( spdx < epdx ); - res =1 ; - } - -return res; -} - -static int __init allocate_cachealigned_memnodemap (void ) -{ -unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); -unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); - - memnodemap =mfn_to_virt (mfn); - mfn <<=PAGE_SHIFT ;xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- size <<=PAGE_SHIFT ; -printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , - mfn, mfn + size); - memnodemapsize = size /sizeof (*memnodemap); - -return 0 ; -} - -/* -* The LSB of all start and end addresses in the node map is the value of the
Quoted 57 lines
xen/arch/x86/numa.c +0
-50-* maximum possible shift. -*/ -static int __init extract_lsb_from_nodes (const struct node *nodes, -int numnodes) -{ -int i, nodes_used =0 ; -unsigned long spdx, epdx; -unsigned long bitfield =0 , memtop =0 ; - -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; - bitfield |= spdx; - nodes_used++; -if ( epdx > memtop ) - memtop = epdx; - } -if ( nodes_used <=1 ) - i =BITS_PER_LONG -1 ; -else - i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); - memnodemapsize = (memtop >> i) +1 ; -return i; -} - -int __init compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids) -{ -int shift; - - shift =extract_lsb_from_nodes (nodes, numnodes); -if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) - memnodemap = _memnodemap; -else if (allocate_cachealigned_memnodemap () ) -return -1 ; -printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); - -if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) - { -printk (KERN_INFO "Your memory is not aligned you need to " -"rebuild your hypervisor with a bigger NODEMAPSIZE " -"shift=%d\n" , shift); -return -1 ; - } - -return shift; -}/* initialize NODE_DATA given nodeid and start/end */ void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) {xen/common/Makefile +0
-0
Quoted 9 lines
xen/common/Makefile obj-y += wait.o · line 57
Quoted 5 lines
xen/common/Makefile +1
-0@@ -54,6 +54,7 @@ obj-y += wait.o obj-bin-y += warning.init.o obj-$(CONFIG_XENOPROF) += xenoprof.o obj-y += xmalloc_tlsf.o +obj-$(CONFIG_NUMA) += numa.oAFAICT, the
Makefileis listing the file in alphabetical order. So please add numa.o in the correct position.
Thanks for the reminder, I will fix it.
Quoted 30 lines
xen/common/Makefile obj-y += wait.o · line 58
Quoted 2 lines
xen/common/Makefile +0
-0obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzmalzo unlzo unlz4 unzstd earlycpio,$(n).init.o)
Quoted 7 lines
xen/common/numa.c new file +0
-0xen/common/numa.c line 10
Quoted 10 lines
xen/common/numa.c new file +10
-0@@ -0,0 +1,131 @@ +/* +* Generic VM initialization for x86-64 NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/mm.h> +#include <xen/string.h> +#include <xen/init.h> +#include <xen/ctype.h> You don't seem to use any helpers./types directly defined by at least this header...
xen/common/numa.c line 13
Quoted 3 lines
xen/common/numa.c new file +3
-0+#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/time.h> ... this one and ...
Quoted 6 lines
xen/common/numa.c line 14
Quoted 1 line
xen/common/numa.c new file +1
-0+#include <xen/smp.h> ... this one. Can you check the list of headers and introduce the minimum? If the dependency is required by another headers, then I think that dependency should be moved in the header requiring it.
I will check it in next version. If it isn't needed, I will remove it.
Quoted 5 lines
xen/common/numa.c line 16
Quoted 2 lines
xen/common/numa.c new file +2
-0+#include <xen/pfn.h> +#include <xen/sched.h> Please sort the includes in alphabetical order.
OK
Quoted 180 lines
Quoted 8 lines
xen/common/numa.c new file +8
-0+ +struct node_data node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +int memnode_shift; +typeof(*memnodemap) _memnodemap[64]; +unsigned long memnodemapsize; +u8 *memnodemap;xen/common/numa.c
Quoted 10 lines
xen/common/numa.c new file +10
-0+ +/* +* Given a shift value, try to populate memnodemap[] +* Returns : +* 1 if OK +* 0 if memnodmap[] too small (of shift too small) +* -1 if node overlap or lost ram (shift too big) +*/ +static int __init populate_memnodemap (const struct node *nodes, + int numnodes, int shift, nodeid_t*nodeids)
xen/common/numa.c
Quoted 5 lines
xen/common/numa.c new file +5
-0+{ +unsigned long spdx, epdx; +int i, res =-1 ; + + memset(memnodemap, NUMA_NO_NODE, memnodemapsize *
sizeof(*memnodemap));Quoted 32 lines
xen/common/numa.c new file +32
-0+for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx ) +continue ; +if ( (epdx >> shift) >= memnodemapsize ) +return 0 ; +do { +if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) +return -1 ; + +if ( !nodeids ) + memnodemap[spdx >> shift] = i; +else + memnodemap[spdx >> shift] = nodeids[i]; + + spdx += (1UL << shift); + }while ( spdx < epdx ); + res =1 ; + } + +return res; +} + +static int __init allocate_cachealigned_memnodemap (void ) +{ +unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); +unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); + + memnodemap =mfn_to_virt (mfn); + mfn <<=PAGE_SHIFT ;xen/common/numa.c
Quoted 10 lines
xen/common/numa.c new file +10
-0+ size <<=PAGE_SHIFT ; +printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , + mfn, mfn + size); + memnodemapsize = size /sizeof (*memnodemap); + +return 0 ; +} + +/* +* The LSB of all start and end addresses in the node map is the value of the
xen/include/asm-x86/numa.h extern int compute_hash_shift(struct node *nodes, int
Quoted 55 lines
xen/common/numa.c new file +50
-0+* maximum possible shift. +*/ +static int __init extract_lsb_from_nodes (const struct node *nodes, +int numnodes) +{ +int i, nodes_used =0 ; +unsigned long spdx, epdx; +unsigned long bitfield =0 , memtop =0 ; + +for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx ) +continue ; + bitfield |= spdx; + nodes_used++; +if ( epdx > memtop ) + memtop = epdx; + } +if ( nodes_used <=1 ) + i =BITS_PER_LONG -1 ; +else + i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); + memnodemapsize = (memtop >> i) +1 ; +return i; +} + +int __init compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids) +{ +int shift; + + shift =extract_lsb_from_nodes (nodes, numnodes); +if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) + memnodemap = _memnodemap; +else if (allocate_cachealigned_memnodemap () ) +return -1 ; +printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); + +if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) + { +printk (KERN_INFO "Your memory is not aligned you need to " +"rebuild your hypervisor with a bigger NODEMAPSIZE " +"shift=%d\n" , shift); +return -1 ; + } + +return shift; +}xen/include/asm-x86/numa.h +0
-0@@ -26,7 +26,6 @@ extern int compute_hash_shift(struct node *nodes, intnumnodes,
Quoted 17 lines
xen/include/asm-x86/numa.h +0
-7extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -#define VIRTUAL_BUG_ON (x)extern void numa_add_cpu (int cpu);extern void numa_init_array (void );@@ -47,34 +46,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -/* Simple perfect hash to map pdx to node numbers */ -extern int memnode_shift; -extern unsigned long memnodemapsize; -extern u8 *memnodemap; - -struct node_data {xen/include/asm-x86/numa.h static inline void clear_node_cpumask(int cpu) · line 65
Quoted 10 lines
xen/include/asm-x86/numa.h +0
-10-unsigned long node_start_pfn; -unsigned long node_spanned_pages; -}; - -extern struct node_data node_data[]; - -static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) -{ -nodeid_t nid; - VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >=memnodemapsize);
Quoted 20 lines
xen/include/asm-x86/numa.h +0
-12- nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; -VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); -return nid; -} - -#define NODE_DATA (nid) (&(node_data[nid])) - -#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) -#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) -#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ - NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node);void srat_parse_regions (u64 addr);xen/include/xen/numa.h +0
-0xen/include/xen/numa.h line 22
Quoted 5 lines
xen/include/xen/numa.h +2
-0@@ -18,4 +18,39 @@ (((d)->vcpu !=NULL && (d)->vcpu[0 ] !=NULL ) \ ?vcpu_to_node ((d)->vcpu[0 ]) :NUMA_NO_NODE ) +/* The following content can be used when NUMA feature is enabled */ +#if defined(CONFIG_NUMA )Please use #ifdef
CONFIG_NUMA
Quoted 10 lines
xen/include/xen/numa.h line 28
Quoted 6 lines
xen/include/xen/numa.h +6
-0+ +/* Simple perfect hash to map pdx to node numbers */ +extern int memnode_shift; +extern unsigned long memnodemapsize; +extern u8 *memnodemap; +extern typeof(*memnodemap) _memnodemap[64];AFAICT, this will be turned static against in a follow-up patch. Can this be avoided?
I will try it in next version.
Quoted 35 lines
xen/include/xen/numa.h
Quoted 13 lines
xen/include/xen/numa.h +13
-0+ +struct node_data { +unsigned long node_start_pfn; +unsigned long node_spanned_pages; +}; + +extern struct node_data node_data[]; +#define VIRTUAL_BUG_ON (x) + +static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) +{ +nodeid_t nid; + VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >=memnodemapsize);
Quoted 6 lines
xen/include/xen/numa.h +5
-0+ nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; +VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); +return nid; +} +xen/include/xen/numa.h line 56
Quoted 10 lines
xen/include/xen/numa.h +9
-0+#define NODE_DATA (nid) (&(node_data[nid])) + +#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) +#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) +#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ + NODE_DATA(nid)->node_spanned_pages) + +#endif /* CONFIG_NUMA */ +#endif /* _XEN_NUMA_H */ Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Julien GrallHi Julien,
Quoted 78 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月23日 18:59
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 17 lines
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41+++++++++++++++++++++++++++++++--
Quoted 38 lines
Quoted 23 lines
Review
Quoted 3 lines
1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 19 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }Quoted 14 lines
xen/arch/arm/numa_device_tree.c line 41
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
Quoted 3 lines
yes, in the very beginning, I want to keep
device_tree_prefix for functions that will handle dtb file. And usedtb_prefix to replace acpi, to indicate, this function is device tree version numa
Quoted 7 lines
implementation.
Thanks for the clarification. The difference between "dtb" and "
device_tree" is quite subttle: the former refers to the binary while the latter refers to the format. Most of the readers are likely to infer they mean the same. So I think this will bring more confusion.
Thanks for the clarification.
Quoted 11 lines
Quoted 3 lines
If that's not the right reason, I will unify all prefix to
device_tree_in next version. How do you think about it?AFAICT, your parsing functions will always start with "
device_tree_parse_". I would prefer if the set replacing the ACPI helpers start with "device_tree_".If you are concern with the length of the function name, then I would suggest to prefix all the functions with "fdt" (We are dealing with the flattened DT after all) or "dt".
That makes sense, I will do it.
Quoted 5 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node to common
In reply to Julien GrallHi Julien,
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月24日 1:54
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 09/40] xen/x86: Movenuma_add_cpu_nodeto commonHi Wei,
Quoted 6 lines · Wei Chen
This function will be reused by Arm later, so we move it from
arch/x86to common. But we keepcpu_to_nodeandnode_to_cpumaskto x86 header file. Becausecpu_to_nodeandnode_to_cpumaskhave different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.
Quoted 4 lines
AFAICT, the Arm helpers are gated by !CONFIG_NUMA and the ones in common code will be gated by
CONFIG_NUMA. So I am not quite too understand why they can't be moved now. Can you clarify it?
Yes, you're right. After we had introduced !CONFIG_NUMA, we can move node_to_cpumask and cpu_to_node in this patch too. I will fix it in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to common
In reply to Wei ChenHi Wei,
Quoted 189 lines · Wei Chen
These data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 50 ------------------------------------- xen/common/numa.c | 51 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 8 ------ xen/include/xen/numa.h | 15 +++++++++++ 4 files changed, 66 insertions(+), 58 deletions(-)xen/arch/x86/srat.c +0
-50@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;static nodemask_t memory_nodes_parsed__initdata ;static nodemask_t processor_nodes_parsed__initdata ; -static struct node nodes[MAX_NUMNODES ]__initdata ;struct pxm2node {unsigned pxm;@@ -37,9 +36,6 @@ static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =static unsigned node_to_pxm (nodeid_t n); -static int num_node_memblks; -static struct node node_memblk_range[NR_NODE_MEMBLKS ]; -static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);static inline bool node_found (unsigned idx,unsigned pxm)@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; - -if (nd->start <= start && nd->end >= end && - memblk_nodeid[i] == node) -return 1 ; - } - -return 0 ; -} - -static __init int conflicting_memblks (u64 start,u64 end) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; -if (nd->start == nd->end) -continue ; -if (nd->end > start && nd->start < end) -return i; -if (nd->end == end && nd->start == start) -return i; - } -return -1 ; -} - -static __init void cutoff_node (int i,u64 start,u64 end) -{ -struct node *nd = &nodes[i]; -if (nd->start < start) { - nd->start = start; -if (nd->end < nd->start) - nd->start = nd->end; - } -if (nd->end > end) { - nd->end = end; -if (nd->start > nd->end) - nd->start = nd->end; - } -} -static __init void bad_srat (void ) {int i;xen/common/numa.c +51
-0@@ -29,6 +29,11 @@ nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ; +struct node nodes[MAX_NUMNODES ]__initdata ; +int num_node_memblks; +struct node node_memblk_range[NR_NODE_MEMBLKS ]; +nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -136,6 +141,52 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,return shift; } +int valid_numa_range (u64 start,u64 end,nodeid_t node) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; + +if (nd->start <= start && nd->end >= end && + memblk_nodeid[i] == node) +return 1 ; + } + +return 0 ; +} + +int __init conflicting_memblks (u64 start,u64 end) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; +if (nd->start == nd->end) +continue ; +if (nd->end > start && nd->start < end) +return i; +if (nd->end == end && nd->start == start) +return i; + } +return -1 ; +} + +void __init cutoff_node (int i,u64 start,u64 end) +{ +struct node *nd = &nodes[i]; +if (nd->start < start) { + nd->start = start; +if (nd->end < nd->start) + nd->start = nd->end; + } +if (nd->end > end) { + nd->end = end; +if (nd->start > nd->end) + nd->start = nd->end; + } +} +void numa_add_cpu (int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]);xen/include/asm-x86/numa.h +0
-8@@ -17,12 +17,6 @@ extern cpumask_t node_to_cpumask[];#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node]))#define node_to_cpumask (node) (node_to_cpumask[node]) -struct node { -u64 start,end; -}; - -extern int compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids);extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );xen/include/xen/numa.h +0
-0@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 64
Quoted 10 lines
xen/include/xen/numa.h +8
-0extern void numa_add_cpu (int cpu); +struct node { +u64 start,end; +}; + +extern struct node nodes[MAX_NUMNODES ]; +extern int num_node_memblks; +extern struct node node_memblk_range[NR_NODE_MEMBLKS ]; +extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];
I am not overly happy that the 4 ariables above are now exported. Looking at the code, they are (only?) used in arch specific code for acpi_numa_memory_affinity_init() and dtb_numa_memory_affinity_init().
There bits touching the variables looks quite similar between the two functions. The main differences seems to be the messages in printk() and the hotplug bits.
So I think we should attempt to abstract the code. IIRC, we discussed some of the way to abstract when Vijay Kilari attempted to add support for NUMA (see [1]). It might be worth to have a look and see if you can re-use some of the ideas.
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 74
Quoted 11 lines
xen/include/xen/numa.h +7
-0+ +extern int compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids); +extern int conflicting_memblks (u64 start,u64 end); +extern void cutoff_node (int i,u64 start,u64 end); +extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
[1] https://lore.kernel.org/xen-devel/[email protected]/
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to common
Hi Julien,
Quoted 219 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 1:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to commonHi Wei,
xen/arch/x86/srat.c static struct pxm2node __read_mostly
Quoted 26 lines · Wei Chen
These data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 50 ------------------------------------- xen/common/numa.c | 51 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 8 ------ xen/include/xen/numa.h | 15 +++++++++++ 4 files changed, 66 insertions(+), 58 deletions(-)xen/arch/x86/srat.c +0
-1@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;static nodemask_t memory_nodes_parsed__initdata ;static nodemask_t processor_nodes_parsed__initdata ; -static struct node nodes[MAX_NUMNODES ]__initdata ;struct pxm2node {unsigned pxm;@@ -37,9 +36,6 @@ static struct pxm2node __read_mostlypxm2node[MAX_NUMNODES] =
xen/common/numa.c int __init compute_hash_shift(struct node *nodes,
Quoted 79 lines
xen/arch/x86/srat.c +0
-49static unsigned node_to_pxm (nodeid_t n); -static int num_node_memblks; -static struct node node_memblk_range[NR_NODE_MEMBLKS ]; -static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);static inline bool node_found (unsigned idx,unsigned pxm)@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; - -if (nd->start <= start && nd->end >= end && - memblk_nodeid[i] == node) -return 1 ; - } - -return 0 ; -} - -static __init int conflicting_memblks (u64 start,u64 end) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; -if (nd->start == nd->end) -continue ; -if (nd->end > start && nd->start < end) -return i; -if (nd->end == end && nd->start == start) -return i; - } -return -1 ; -} - -static __init void cutoff_node (int i,u64 start,u64 end) -{ -struct node *nd = &nodes[i]; -if (nd->start < start) { - nd->start = start; -if (nd->end < nd->start) - nd->start = nd->end; - } -if (nd->end > end) { - nd->end = end; -if (nd->start > nd->end) - nd->start = nd->end; - } -} -static __init void bad_srat (void ) {int i;xen/common/numa.c +5
-0@@ -29,6 +29,11 @@ nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ; +struct node nodes[MAX_NUMNODES ]__initdata ; +int num_node_memblks; +struct node node_memblk_range[NR_NODE_MEMBLKS ]; +nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -136,6 +141,52 @@ int __init compute_hash_shift(struct node *nodes,int numnodes,
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t
Quoted 83 lines
xen/common/numa.c +46
-0return shift; } +int valid_numa_range (u64 start,u64 end,nodeid_t node) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; + +if (nd->start <= start && nd->end >= end && + memblk_nodeid[i] == node) +return 1 ; + } + +return 0 ; +} + +int __init conflicting_memblks (u64 start,u64 end) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; +if (nd->start == nd->end) +continue ; +if (nd->end > start && nd->start < end) +return i; +if (nd->end == end && nd->start == start) +return i; + } +return -1 ; +} + +void __init cutoff_node (int i,u64 start,u64 end) +{ +struct node *nd = &nodes[i]; +if (nd->start < start) { + nd->start = start; +if (nd->end < nd->start) + nd->start = nd->end; + } +if (nd->end > end) { + nd->end = end; +if (nd->start > nd->end) + nd->start = nd->end; + } +} +void numa_add_cpu (int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]);xen/include/asm-x86/numa.h +0
-8@@ -17,12 +17,6 @@ extern cpumask_t node_to_cpumask[];#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node]))#define node_to_cpumask (node) (node_to_cpumask[node]) -struct node { -u64 start,end; -}; - -extern int compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids);extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );xen/include/xen/numa.h +0
-0@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t
phys_to_nid(paddr_t addr)xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 64
Quoted 11 lines
xen/include/xen/numa.h +8
-0extern void numa_add_cpu (int cpu); +struct node { +u64 start,end; +}; + +extern struct node nodes[MAX_NUMNODES ]; +extern int num_node_memblks; +extern struct node node_memblk_range[NR_NODE_MEMBLKS ]; +extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];I am not overly happy that the 4 ariables above are now exported. Looking at the code, they are (only?) used in arch specific code for
acpi_numa_memory_affinity_init()anddtb_numa_memory_affinity_init().
Quoted 8 lines
There bits touching the variables looks quite similar between the two functions. The main differences seems to be the messages in
printk()and the hotplug bits.So I think we should attempt to abstract the code. IIRC, we discussed some of the way to abstract when Vijay Kilari attempted to add support for NUMA (see [1]). It might be worth to have a look and see if you can re-use some of the ideas.
Ok, I will look at that thread. If it's useful, I will do it in next version.
Quoted 19 lines
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
Quoted 12 lines
xen/include/xen/numa.h +7
-0+ +extern int compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids); +extern int conflicting_memblks (u64 start,u64 end); +extern void cutoff_node (int i,u64 start,u64 end); +extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */ [1]
https://lore.kernel.org/xen-devel/1500378106-2620-1-git-send-email-
[email protected]/--
Julien Grall
Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common
In reply to Wei ChenHi Wei,
Quoted 8 lines · Wei Chen
This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem,numa_init_arrayandnuma_emulation.As
numa_initmem_inithas been moved to common,_memnodemapis not used cross files. We can restore_memnodemapto static.
As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.
Quoted 151 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)xen/arch/x86/numa.c +0
-118@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn; - -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1 cpu->node -mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ -if (hweight64 (sz) >1 ) - { -u64 x =1 ; -while ( (x <<1 ) < sz ) - x <<=1 ; -if ( x < sz/2 ) -printk (KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n" ); - sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); -for ( i =0 ; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; -if ( i == numa_fake -1 ) - sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; -printk (KERN_INFO "Faking node %d at %" PRIx64"-%" PRIx64" (%" PRIu64"MB)\n" , - i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >>20 ); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); -if ( memnode_shift <0 ) - { - memnode_shift =0 ; -printk (KERN_ERR "No NUMA hash function found. Emulation disabled.\n" ); -return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) -{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ; -#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - -printk (KERN_INFO "%s\n" , - numa_off ?"NUMA turned off" :"No NUMA configuration found" ); - -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;xen/common/numa.c +0
-0
xen/common/numa.c line 16
Quoted 3 lines
xen/common/numa.c +0
-0@@ -14,12 +14,13 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h>
NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.
Quoted 21 lines
xen/common/numa.c +4
-1+#include <asm/acpi.h> struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ int memnode_shift; -typeof(*memnodemap) _memnodemap[64]; +static typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap;@@ -34,6 +35,8 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +bool numa_off; +/* * Given a shift value, try to populate memnodemap[] * Returns :
xen/common/numa.c void numa_add_cpu(int cpu) · line 199
Quoted 6 lines
xen/common/numa.c +3
-0@@ -191,3 +194,120 @@ void numa_add_cpu(int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } + +/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end)
From an abstract PoV, start and end should be paddr_t. This should be done on a separate patch though.
Quoted 26 lines
xen/common/numa.c +26
-0+{ +unsigned long start_pfn, end_pfn; + + start_pfn = start >>PAGE_SHIFT ; + end_pfn = end >>PAGE_SHIFT ; + +NODE_DATA (nodeid)->node_start_pfn = start_pfn; +NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn; + +node_set_online (nodeid); +} + +void __init numa_init_array (void ) +{ +int rr, i; + +/* There are unfortunately some poorly designed mainboards around +that only connect memory to a single CPU. This breaks the 1:1 cpu->node +mapping. To avoid this fill in the mapping for all possible +CPUs, as the number of CPUs is not known yet. +We round robin the existing nodes. */ + rr =first_node (node_online_map); +for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] !=NUMA_NO_NODE ) +continue ;
xen/common/numa.c void numa_add_cpu(int cpu) · line 235
Quoted 10 lines
xen/common/numa.c +10
-0+numa_set_node (i, rr); + rr =cycle_node (rr, node_online_map); + } +} + +#ifdef CONFIG_NUMA_EMU +int numa_fake__initdata =0 ; + +/* Numa emulation */ +static int __init numa_emulation (u64 start_pfn,u64 end_pfn)
Here, this should be either "unsigned long" or ideally "mfn_t". Although, if you use "unsigned long", you will need to...
xen/common/numa.c void numa_add_cpu(int cpu) · line 239
Quoted 4 lines
xen/common/numa.c +4
-0+{ +int i; +struct node nodes[MAX_NUMNODES ]; +u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake;
... cast "(end_pfn - start_pfn)" to uin64_t or use pfn_to_paddr().
Quoted 42 lines
xen/common/numa.c +42
-0+ +/* Kludge needed for the hash function */ +if (hweight64 (sz) >1 ) + { +u64 x =1 ; +while ( (x <<1 ) < sz ) + x <<=1 ; +if ( x < sz/2 ) +printk (KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n" ); + sz = x; + } + +memset (&nodes,0 ,sizeof (nodes)); +for ( i =0 ; i < numa_fake; i++ ) + { + nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; +if ( i == numa_fake -1 ) + sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; + nodes[i].end = nodes[i].start + sz; +printk (KERN_INFO "Faking node %d at %" PRIx64"-%" PRIx64" (%" PRIu64"MB)\n" , + i, + nodes[i].start, nodes[i].end, + (nodes[i].end - nodes[i].start) >>20 ); +node_set_online (i); + } + memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); +if ( memnode_shift <0 ) + { + memnode_shift =0 ; +printk (KERN_ERR "No NUMA hash function found. Emulation disabled.\n" ); +return -1 ; + } +for_each_online_node ( i ) + setup_node_bootmem(i, nodes[i].start, nodes[i].end); +numa_init_array (); + + return 0; +} +#endif + +void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) +{
xen/common/numa.c void numa_add_cpu(int cpu) · line 291
Quoted 10 lines
xen/common/numa.c +10
-0+int i; + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_ACPI_NUMA +if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ) )
(u64)v << PAGE_SHIFT should be switched to use pfn_to_paddr() or mfn_to_paddr() if you decide to make start_pfn and end_pfn typesafe.
xen/common/numa.c void numa_add_cpu(int cpu) · line 300
Quoted 9 lines
xen/common/numa.c +9
-0+return ; +#endif + +printk (KERN_INFO "%s\n" , + numa_off ?"NUMA turned off" :"No NUMA configuration found" ); + +printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , + (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT );
Same remark here. PRIx64 would also have to be switched to PRIpaddr.
Quoted 63 lines
xen/common/numa.c +13
-0+/* setup dummy node covering all memory */ + memnode_shift =BITS_PER_LONG -1 ; + memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap); + +nodes_clear (node_online_map); +node_set_online (0 ); +for ( i =0 ; i < nr_cpu_ids; i++ ) +numa_set_node (i,0 ); +cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); +setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ); +}xen/include/asm-x86/numa.h +0
-5@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_init_array (void ); -extern bool numa_off; - -extern int srat_disabled (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );xen/include/asm-x86/setup.h +0
-1@@ -17,7 +17,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );xen/include/xen/numa.h +0
-1@@ -26,7 +26,6 @@extern int memnode_shift;extern unsigned long memnodemapsize;extern u8 *memnodemap; -extern typeof(*memnodemap) _memnodemap[64];struct node_data {unsigned long node_start_pfn;@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node);
xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 80
Quoted 10 lines
xen/include/xen/numa.h +7
-0+extern void numa_init_array (void ); +extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern bool numa_off; +extern int numa_fake; + +extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
In reply to Wei ChenHi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.
IHMO numa_set_node() should behave exactly the same way on x86 and Arm because this is going to be used by the common code.
From the commit message, I don't quite understand why the check is necessary on Arm but not on x86. Can you clarify it?
Quoted 50 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa.c | 31 +++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-)create mode 100644
xen/arch/arm/numa.cxen/arch/arm/Makefile +1
-0@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o +obj-$(CONFIG_NUMA) += numa.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.oxen/arch/arm/numa.c new file +21
-0@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h>
xen/arch/arm/numa.c line 31
Quoted 10 lines
xen/arch/arm/numa.c new file +10
-0+#include <xen/numa.h> + +void numa_set_node (int cpu,nodeid_t nid) +{ +if ( nid >=MAX_NUMNODES || + !nodemask_test (nid, &node_online_map) ) + nid =0 ; + + cpu_to_node[cpu] = nid; +}
I think numa_set_node() will want to be implemented in common code.
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 30
Quoted 9 lines
xen/include/asm-arm/numa.h +1
-0@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_set_node (x, y) do { } while (0)
I would define it in xen/numa.h so other arch can take advantage ot it. Also, please use a static inline helper so the arguments are evaluated.
Quoted 22 lines
xen/include/asm-arm/numa.h +1
-0+#endif #endif /* __ARCH_ARM_NUMA_H */ xen/include/asm-x86/numa.h +0
-1@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))extern int srat_disabled (void ); -extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, nodeid_t node); · line 76
Quoted 7 lines
xen/include/xen/numa.h +1
-0@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern void numa_set_node (int cpu,nodeid_t node);extern bool numa_off;extern int numa_fake;
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common
In reply to Julien GrallHi Julien,
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 12/40] xen/x86: Movenuma_initmem_initto commonHi Wei,
Quoted 5 lines · Wei Chen
This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem,numa_init_arrayandnuma_emulation.
Quoted 7 lines
Quoted 3 lines
As
numa_initmem_inithas been moved to common,_memnodemapis not used cross files. We can restore_memnodemapto static.As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.
Yes, thanks!
Quoted 155 lines
Quoted 36 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)xen/arch/x86/numa.c +0
-11@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/arch/x86/numa.c int srat_disabled(void) · line 68
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 21 lines
xen/arch/x86/numa.c +0
-21-mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ];xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ - if ( hweight64(sz) > 1 ) - { - u64 x = 1; - while ( (x << 1) < sz ) - x <<= 1; - if ( x < sz/2 ) - printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11- sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); - for ( i = 0; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; - if ( i == numa_fake - 1 ) - sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; - printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >> 20); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); - if ( memnode_shift < 0 ) - { - memnode_shift = 0; - printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11-return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 6 lines
xen/arch/x86/numa.c +0
-6-{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ;xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - - printk(KERN_INFO "%s\n", - numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 24 lines
xen/arch/x86/numa.c +0
-18- -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;xen/common/numa.c +0
-1--- a/xen/common/numa.c
Quoted 9 lines
xen/common/numa.c line 16
Quoted 5 lines
+++
b/xen/common/numa.cxen/common/numa.c +0
-0@@ -14,12 +14,13 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.
got it
Quoted 15 lines
Quoted 15 lines
xen/common/numa.c +2
-1+#include <asm/acpi.h> struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ int memnode_shift; -typeof(*memnodemap) _memnodemap[64]; +static typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap;@@ -34,6 +35,8 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];
Quoted 16 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 199
Quoted 12 lines
xen/common/numa.c +5
-0+bool numa_off; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -191,3 +194,120 @@ void numa_add_cpu(int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } + +/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end)From an abstract PoV, start and end should be
paddr_t. This should be
done on a separate patch though.
Ok.
Quoted 32 lines
Quoted 8 lines
xen/common/numa.c +8
-0+{ +unsigned long start_pfn, end_pfn; + + start_pfn = start >>PAGE_SHIFT ; + end_pfn = end >>PAGE_SHIFT ; + +NODE_DATA (nodeid)->node_start_pfn = start_pfn; +NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ +node_set_online (nodeid); +} + +void __init numa_init_array (void ) +{ +int rr, i; + +/* There are unfortunately some poorly designed mainboards around +that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 13 lines
xen/common/numa.c +13
-0+mapping. To avoid this fill in the mapping for all possible +CPUs, as the number of CPUs is not known yet. +We round robin the existing nodes. */ + rr =first_node (node_online_map); +for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] !=NUMA_NO_NODE ) +continue ; +numa_set_node (i, rr); + rr =cycle_node (rr, node_online_map); + } +} +
Quoted 9 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 235
Quoted 5 lines
xen/common/numa.c +5
-0+#ifdef CONFIG_NUMA_EMU +int numa_fake__initdata =0 ; + +/* Numa emulation */ +static int __init numa_emulation (u64 start_pfn,u64 end_pfn)Here, this should be either "unsigned long" or ideally "
mfn_t". Although, if you use "unsigned long", you will need to...
Do we need a separate patch to do it?
Quoted 7 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 239
Quoted 4 lines
xen/common/numa.c +4
-0+{ +int i; +struct node nodes[MAX_NUMNODES ]; +u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake;... cast "(
end_pfn-start_pfn)" touin64_tor usepfn_to_paddr().
Ok
Quoted 45 lines
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 9 lines
xen/common/numa.c +9
-0+ +/* Kludge needed for the hash function */ + if ( hweight64(sz) > 1 ) + { + u64 x = 1; + while ( (x << 1) < sz ) + x <<= 1; + if ( x < sz/2 ) + printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+ sz = x; + } + +memset (&nodes,0 ,sizeof (nodes)); + for ( i = 0; i < numa_fake; i++ ) + { + nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; + if ( i == numa_fake - 1 ) + sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; + nodes[i].end = nodes[i].start + sz; + printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ i, + nodes[i].start, nodes[i].end, + (nodes[i].end - nodes[i].start) >> 20); +node_set_online (i); + } + memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); + if ( memnode_shift < 0 ) + { + memnode_shift = 0; + printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+return -1 ; + } +for_each_online_node ( i ) + setup_node_bootmem(i, nodes[i].start, nodes[i].end); +numa_init_array (); + +return 0 ; +} +#endif + +void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)
Quoted 15 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 291
Quoted 11 lines
xen/common/numa.c +11
-0+{ +int i; + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_ACPI_NUMA +if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ) )(u64)v <<
PAGE_SHIFTshould be switched to usepfn_to_paddr()ormfn_to_paddr()if you decide to makestart_pfnandend_pfntypesafe.
Still need a separate patch to change it before move?
Quoted 6 lines
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 5 lines
xen/common/numa.c +5
-0+return ; +#endif + + printk(KERN_INFO "%s\n", + numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 7 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 300
Quoted 4 lines
xen/common/numa.c +4
-0+ +printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , + (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT );Same remark here. PRIx64 would also have to be switched to PRIpaddr.
Hmm, It seems I'd better to use a separate patch to do PRIpaddr clean up before move code.
Quoted 79 lines
Quoted 59 lines
xen/common/numa.c +13
-0+/* setup dummy node covering all memory */ + memnode_shift =BITS_PER_LONG -1 ; + memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap); + +nodes_clear (node_online_map); +node_set_online (0 ); +for ( i =0 ; i < nr_cpu_ids; i++ ) +numa_set_node (i,0 ); +cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); +setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ); +}xen/include/asm-x86/numa.h +0
-5@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_init_array (void ); -extern bool numa_off; - -extern int srat_disabled (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );xen/include/asm-x86/setup.h +0
-1@@ -17,7 +17,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );xen/include/xen/numa.h +0
-1@@ -26,7 +26,6 @@extern int memnode_shift;extern unsigned long memnodemapsize;extern u8 *memnodemap; -extern typeof(*memnodemap) _memnodemap[64];struct node_data {unsigned long node_start_pfn;xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 72
Quoted 5 lines
xen/include/xen/numa.h +2
-0@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern void numa_init_array (void ); +extern void numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn);xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 80
Quoted 9 lines
xen/include/xen/numa.h +5
-0+extern bool numa_off; +extern int numa_fake; + +extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */ Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
In reply to Julien GrallHi Julien,
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmHi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.
Quoted 6 lines
IHMO
numa_set_node()should behave exactly the same way on x86 and Arm because this is going to be used by the common code.From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?
Yes, in patch#27, in smpboot.c, dt_smp_init_cpus function. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invoke numa_set_node to create CPU-NODE map. But in our testing, we found when NUMA init failed, numa_set_node still can set CPU to a offline or invalid NODE. So we're using node_online_map to prevent this behavior. Otherwise we have to check node_online_map everywhere before we call numa_set_node.
x86 actually is doing the same way, but it handles node_online_map check out of numa_set_node:
57 void __init init_cpu_to_node(void)
58 {
59 unsigned int i;
60 nodeid_t node;
61
62 for ( i = 0; i < nr_cpu_ids; i++ )
63 {
64 u32 apicid = x86_cpu_to_apicid[i];
65 if ( apicid == BAD_APICID )
66 continue;
67 node = apicid < MAX_LOCAL_APIC ? apicid_to_node[apicid] : NUMA_NO_NODE;
68 if ( node == NUMA_NO_NODE || !node_online(node) )
69 node = 0;
70 numa_set_node(i, node);
71 }
72 }
Quoted 53 lines
Quoted 53 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa.c | 31 +++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-)create mode 100644
xen/arch/arm/numa.cxen/arch/arm/Makefile +1
-0@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o +obj-$(CONFIG_NUMA) += numa.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.oxen/arch/arm/numa.c new file +24
-0@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +void numa_set_node (int cpu,nodeid_t nid)
Quoted 9 lines
xen/arch/arm/numa.c line 31
Quoted 7 lines
xen/arch/arm/numa.c new file +7
-0+{ +if ( nid >=MAX_NUMNODES || + !nodemask_test (nid, &node_online_map) ) + nid =0 ; + + cpu_to_node[cpu] = nid; +}I think
numa_set_node()will want to be implemented in common code.
See my above comment. If x86 is ok, I think yes, we can do it in common code.
Quoted 8 lines
Quoted 8 lines
xen/include/asm-arm/numa.h +0
-0@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20)
Quoted 5 lines
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 30
Quoted 1 line
xen/include/asm-arm/numa.h +1
-0+#define numa_set_node (x, y) do { } while (0)I would define it in
xen/numa.hso other arch can take advantage ot it. Also, please use a static inline helper so the arguments are evaluated.
Ok
Quoted 36 lines
xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end,
Quoted 21 lines
xen/include/asm-arm/numa.h +1
-0+#endif #endif /* __ARCH_ARM_NUMA_H */ xen/include/asm-x86/numa.h +0
-1@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))extern int srat_disabled (void ); -extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);xen/include/xen/numa.h +0
-0@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end,
nodeid_tnode);xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, · line 71
Quoted 3 lines
xen/include/xen/numa.h +0
-0extern void numa_init_array (void ); extern void numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn);xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
Quoted 5 lines
xen/include/xen/numa.h +1
-0+extern void numa_set_node (int cpu,nodeid_t node);extern bool numa_off;extern int numa_fake;Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 26 lines
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmHi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.Quoted 6 lines
IHMO
numa_set_node()should behave exactly the same way on x86 and Arm because this is going to be used by the common code.From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?
Quoted 8 lines
Yes, in patch#27, in
smpboot.c,dt_smp_init_cpusfunction. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invokenuma_set_nodeto create CPU-NODE map. But in our testing, we found when NUMA init failed,numa_set_nodestill can set CPU to a offline or invalid NODE. So we're usingnode_online_mapto prevent this behavior. Otherwise we have to checknode_online_mapeverywhere before we callnuma_set_node.
What do you mean by invalid NODE? Is it 0xFF (NUMA_NO_NODE)?
Quoted 3 lines
x86 actually is doing the same way, but it handles
node_online_mapcheck out ofnuma_set_node:
Right...
Quoted 5 lines
Quoted 2 lines
I think
numa_set_node()will want to be implemented in common code.See my above comment. If x86 is ok, I think yes, we can do it in common code.
... on x86, this check is performed outside of numa_set_node() for one caller whereas on Arm you are adding it in numa_set_node().
For example, numa_set_node() can be called with NUMA_NO_NODE. On x86, we would set cpu_to_node[] to that value. However, if I am not mistaken, on Arm we would set the value to 0.
This will change the behavior of users to cpu_to_node() later on (such as XEN_SYSCTL_cputopoinfo).
NUMA is not something architecture specific, so I dont't think the implementation should differ here.
In this case, I think numa_set_node() shouldn't check if the node is valid. Instead, the caller should take care of it if it is important.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 270 lines
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 12/40] xen/x86: Movenuma_initmem_initto commonHi Wei,
Quoted 5 lines · Wei Chen
This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem,numa_init_arrayandnuma_emulation.Quoted 7 lines
Quoted 3 lines
As
numa_initmem_inithas been moved to common,_memnodemapis not used cross files. We can restore_memnodemapto static.As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.
Yes, thanks!
Quoted 155 lines
Quoted 36 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)xen/arch/x86/numa.c +0
-11@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/arch/x86/numa.c int srat_disabled(void) · line 68
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 21 lines
xen/arch/x86/numa.c +0
-21-mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ];xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ - if ( hweight64(sz) > 1 ) - { - u64 x = 1; - while ( (x << 1) < sz ) - x <<= 1; - if ( x < sz/2 ) - printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11- sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); - for ( i = 0; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; - if ( i == numa_fake - 1 ) - sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; - printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >> 20); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); - if ( memnode_shift < 0 ) - { - memnode_shift = 0; - printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11-return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 6 lines
xen/arch/x86/numa.c +0
-6-{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ;xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - - printk(KERN_INFO "%s\n", - numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 24 lines
xen/arch/x86/numa.c +0
-18- -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;xen/common/numa.c +0
-1--- a/xen/common/numa.cQuoted 9 lines
xen/common/numa.c line 16
Quoted 5 lines
+++
b/xen/common/numa.cxen/common/numa.c +0
-0@@ -14,12 +14,13 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.
got it
Quoted 15 lines
Quoted 15 lines
xen/common/numa.c +2
-1+#include <asm/acpi.h> struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ int memnode_shift; -typeof(*memnodemap) _memnodemap[64]; +static typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap;@@ -34,6 +35,8 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];Quoted 16 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 199
Quoted 12 lines
xen/common/numa.c +5
-0+bool numa_off; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -191,3 +194,120 @@ void numa_add_cpu(int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } + +/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end)From an abstract PoV, start and end should be
paddr_t. This should be
done on a separate patch though.Ok.
Quoted 32 lines
Quoted 8 lines
xen/common/numa.c +8
-0+{ +unsigned long start_pfn, end_pfn; + + start_pfn = start >>PAGE_SHIFT ; + end_pfn = end >>PAGE_SHIFT ; + +NODE_DATA (nodeid)->node_start_pfn = start_pfn; +NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ +node_set_online (nodeid); +} + +void __init numa_init_array (void ) +{ +int rr, i; + +/* There are unfortunately some poorly designed mainboards around +that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 13 lines
xen/common/numa.c +13
-0+mapping. To avoid this fill in the mapping for all possible +CPUs, as the number of CPUs is not known yet. +We round robin the existing nodes. */ + rr =first_node (node_online_map); +for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] !=NUMA_NO_NODE ) +continue ; +numa_set_node (i, rr); + rr =cycle_node (rr, node_online_map); + } +} +Quoted 9 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 235
Quoted 5 lines
xen/common/numa.c +5
-0+#ifdef CONFIG_NUMA_EMU +int numa_fake__initdata =0 ; + +/* Numa emulation */ +static int __init numa_emulation (u64 start_pfn,u64 end_pfn)Here, this should be either "unsigned long" or ideally "
mfn_t". Although, if you use "unsigned long", you will need to...
Quoted 1 line
Do we need a separate patch to do it?
I would prefer if the cleanups are done separately as this makes easier to review code movement.
Quoted 72 lines
Quoted 8 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 239
Quoted 5 lines
xen/common/numa.c +4
-0+{ +int i; +struct node nodes[MAX_NUMNODES ]; +u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake;... cast "(
end_pfn-start_pfn)" touin64_tor usepfn_to_paddr().Ok
Quoted 45 lines
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 9 lines
xen/common/numa.c +9
-0+ +/* Kludge needed for the hash function */ + if ( hweight64(sz) > 1 ) + { + u64 x = 1; + while ( (x << 1) < sz ) + x <<= 1; + if ( x < sz/2 ) + printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+ sz = x; + } + +memset (&nodes,0 ,sizeof (nodes)); + for ( i = 0; i < numa_fake; i++ ) + { + nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; + if ( i == numa_fake - 1 ) + sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; + nodes[i].end = nodes[i].start + sz; + printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ i, + nodes[i].start, nodes[i].end, + (nodes[i].end - nodes[i].start) >> 20); +node_set_online (i); + } + memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); + if ( memnode_shift < 0 ) + { + memnode_shift = 0; + printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+return -1 ; + } +for_each_online_node ( i ) + setup_node_bootmem(i, nodes[i].start, nodes[i].end); +numa_init_array (); + +return 0 ; +} +#endif + +void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 15 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 291
Quoted 11 lines
xen/common/numa.c +11
-0+{ +int i; + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_ACPI_NUMA +if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ) )(u64)v <<
PAGE_SHIFTshould be switched to usepfn_to_paddr()ormfn_to_paddr()if you decide to makestart_pfnandend_pfntypesafe.
Quoted 1 line
Still need a separate patch to change it before move?
Yes.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default
In reply to Wei ChenHi Wei,
Quoted 13 lines · Wei Chen
Today's Arm64 systems can reach or exceed 16 NUMA nodes, so we set the number to 64 to match with x86.
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)xen/include/asm-arm/numa.h +0
-0
xen/include/asm-arm/numa.h line 13
Quoted 10 lines
xen/include/asm-arm/numa.h +6
-1@@ -5,7 +5,15 @@typedef u8 nodeid_t ; -#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/
Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to xen/numa.h.
xen/include/asm-arm/numa.h line 19
Quoted 7 lines
xen/include/asm-arm/numa.h +3
-0+#define NODES_SHIFT 6 + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default
Quoted 7 lines · Julien Grall
Quoted 6 lines · Wei Chen
xen/include/asm-arm/numa.h +0
-0@@ -5,7 +5,15 @@typedef u8 nodeid_t ;
Quoted 11 lines
xen/include/asm-arm/numa.h line 13
Quoted 7 lines
xen/include/asm-arm/numa.h +6
-1-#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/ Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to
xen/numa.h.
To be honest - if this gets moved, please at least consider making it a proper Kconfig setting. Just as much a the number of CPUs can be configured, the number of nodes should be possible to choose by the build manager. Of course - if it's not too much trouble ...
Jan
Re: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type
In reply to Wei ChenHi Wei,
Quoted 9 lines · Wei Chen
In later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce
fdt_node_check_typein this patch to avoid redundant code in subsequent patches.Signed-off-by Wei Chen <[email protected]>--- xen/common/libfdt/fdt_ro.c | 15 +++++++++++++++ xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++
This is meant to be a verbatim copy of libfdt. So I am not entirely in favor of adding a new function therefore without been upstreamed to libfdt first.
Quoted 10 lines
2 files changed, 40 insertions(+)xen/common/libfdt/fdt_ro.c +0
-0@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, int nodeoffset,return 1 ; }
xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int nodeoffset, · line 557
Quoted 10 lines
xen/common/libfdt/fdt_ro.c +10
-0+int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type) +{ +const void *prop; +int len; + + prop =fdt_getprop (fdt, nodeoffset,"device_type" , &len); +if (!prop) +return len; +if (fdt_stringlist_contains (prop, len, type))
The "device_type" is not a list of string. So I am a bit confused why you are using this helper. Shouldn't we simply check that the property value and type matches?
Quoted 35 lines
xen/common/libfdt/fdt_ro.c +5
-0+return 0 ; + else + return 1; +} +int fdt_node_offset_by_compatible (const void *fdt,int startoffset,const char *compatible) {xen/include/xen/libfdt/libfdt.h +18
-0@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);int fdt_node_check_compatible (const void *fdt,int nodeoffset,const char *compatible); +/** +* fdt_node_check_type: check a node's device_type property +* @fdt: pointer to the device tree blob +* @nodeoffset: offset of a tree node +* @type: string to match against +* +* +* fdt_node_check_type() returns 0 if the given node contains a 'device_type' +* property with the given string as one of its elements, it returns non-zero +* otherwise, or on error. +* +* returns: +* 0, if the node has a 'device_type' property listing the given string +* 1, if the node has a 'device_type' property, but it does not list +* the given string +* -FDT_ERR_NOTFOUND, if the given node has no 'device_type' property +* -FDT_ERR_BADOFFSET, if nodeoffset does not refer to a BEGIN_NODE tag +* -FDT_ERR_BADMAGIC,
xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle); · line 829
Quoted 10 lines
xen/include/xen/libfdt/libfdt.h +7
-0+* -FDT_ERR_BADVERSION, +* -FDT_ERR_BADSTATE, +* -FDT_ERR_BADSTRUCTURE, standard meanings +*/ +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type); +/** * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value * @fdt: pointer to the device tree blob
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
In reply to Wei ChenHi Wei,
Quoted 31 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +4
-0@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{
The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; + + end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat ();
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 86
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */ + i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) {
Xen coding style is using:
if ( ... )
{
Note that I may not comment on all the occurents, so please check the other places.
Quoted 50 lines
xen/arch/arm/numa_device_tree.c +46
-0+/* +* Overlaps with other memblk in the same node, warning here. +* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 +"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , + node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++; + +return 0 ; +} +/* Parse CPU NUMA node info */ int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) {
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 152
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +5
-0@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)return dtb_numa_processor_affinity_init (nid); } + +/* Parse memory node NUMA info */ +int __init +device_tree_parse_numa_memory_node (const void *fdt,int node, +const char *name,uint32_t addr_cells,uint32_t size_cells)
This is pretty much a copy of process_memory_node(). Can we consider to collect the NUMA ID from there? If not, can we at least abstract the code?
Quoted 26 lines
xen/arch/arm/numa_device_tree.c +26
-0+{ +uint32_t nid; +int ret =0 , len; +paddr_t addr, size; +const struct fdt_property *prop; +uint32_t idx, ranges; +const __be32 *addresses; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + + prop =fdt_get_property (fdt, node,"reg" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"fdt: node `%s': missing `reg' property\n" , name); +return -EINVAL ; + } + + addresses = (const __be32 *)prop->data; + ranges = len / (sizeof (__be32)* (addr_cells + size_cells)); +for ( idx =0 ; idx < ranges; idx++ )
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 188
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ { +device_tree_get_reg (&addresses, addr_cells, size_cells, &addr, &size); +/* Skip zero size ranges */ +if ( !size ) +continue ; + + ret =dtb_numa_memory_affinity_init (nid, addr, size); +if ( ret ) { + printk(XENLOG_WARNING + "NUMA: process range#%d addr = %lx size=%lx failed!\n",
s/%d/%u/ as idx is an unsigned int
s/%lx/%"PRI_paddr"/ as addr and size are paddr_t.
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +4
-0+ idx, addr, size); +return -EINVAL ; + } + }
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 202
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ +if ( idx ==0 ) + { +printk (XENLOG_ERR +"bad property in memory node, idx=%d ret=%d\n" , idx, ret); +return -EINVAL ; + } + +return 0 ; +}
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
In reply to Wei ChenHi Wei,
Quoted 2 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a
s/introduce/introduces/
Quoted 13 lines
new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 210
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +8
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;
entry_count and i should be unsigned. len unfortunately can't because fdt_get_property expects a signed int.
Quoted 11 lines
xen/arch/arm/numa_device_tree.c +11
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + }
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 231
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 )
I understand that entry_count may be 0. But I can't see how it can be negative as the property len cannot be (even if it is a signed type). So I think this wants to be "== 0".
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 244
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+ { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;
You can use dt_next_cell() which will update the pointer for you.
Quoted 16 lines
xen/arch/arm/numa_device_tree.c +15
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + +if ( (from == to && distance !=NUMA_LOCAL_DISTANCE ) || + (from != to && distance <=NUMA_LOCAL_DISTANCE ) ) + { +printk (XENLOG_WARNING +"Invalid nodes' distance from node#%d to node#%d = %d\n" , + from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 269
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance); + } + +return 0 ; +}
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
In reply to Wei ChenHi Wei,
Quoted 36 lines · Wei Chen
When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/setup.c | 6 ++++++ xen/arch/arm/smpboot.c | 6 ++++++ xen/include/asm-arm/numa.h | 1 + 3 files changed, 13 insertions(+)xen/arch/arm/setup.c +6
-0@@ -918,6 +918,12 @@ void __init start_xen(unsigned long boot_phys_offset,processor_id (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (0 ); +smp_init_cpus (); cpus =smp_get_max_cpus ();printk (XENLOG_INFO "SMP: Allowing %u CPUs\n" , cpus);xen/arch/arm/smpboot.c +0
-0
xen/arch/arm/smpboot.c void start_secondary(void) · line 366
Quoted 9 lines
xen/arch/arm/smpboot.c +6
-0@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +
On x86, numa_add_cpu() will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 62
Quoted 12 lines
xen/arch/arm/smpboot.c +0
-0/* Now report this CPU is up */ cpumask_set_cpu (cpuid, &cpu_online_map);xen/include/asm-arm/numa.h +1
-0@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;#define __node_distance (a, b) (20)#define numa_init (x) do { } while (0) +#define numa_add_cpu (x) do { } while (0)
This is a stubs for a common helper. So I think this wants to be moved in the !CONFIG_NUMA in xen/numa.h.
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 65
Quoted 4 lines
xen/include/asm-arm/numa.h +0
-0#define numa_set_node (x, y) do { } while (0)#endif
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map
In reply to Wei ChenHi Wei,
Quoted 3 lines · Wei Chen
Sometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.
This commit message gives the impression that you are trying to fix a bug. However, what you are explaining is the reason why the code will use the logical ID rather than physical ID.
I think the commit message should explain what the patch is doing. You can then add an explanation why you are using the CPU logical ID. Something like "Note we storing the CPU logical ID because...".
Quoted 24 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)xen/arch/arm/smpboot.c +5
-0@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void) { [0 ... NR_CPUS - 1] = MPIDR_INVALID }; +static nodeid_t node_map[NR_CPUS ]__initdata = + { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE + };bool bootcpu_valid = false; +uint32_t nid =0 ;int rc; mpidr = boot_cpu_data.mpidr.bits &MPIDR_HWID_MASK ;
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 185
Quoted 9 lines
xen/arch/arm/smpboot.c +6
-0@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)continue ; } +#ifdef CONFIG_DEVICE_TREE_NUMA +/* +* When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation +* from CPU dts node, otherwise the nid is always 0. +*/ +if ( !dt_property_read_u32 (cpu,"numa-node-id" , &nid) )
You can avoid the #ifdef by writing:
if ( IS_ENABLED(CONFIG_DEVICE_TREE_NUMA) && ... )
However, I would using CONFIG_NUMA because this code is already DT specific. So we can shorten the name a bit.
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 188
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+ { + printk(XENLOG_WARNING + "cpu[%d] dts path: %s: doesn't have numa infomation!\n",
s/information/information/
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 191
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+ cpuidx, dt_node_full_name(cpu)); +/* +* The the early stage of NUMA initialization, when Xen found any
s/The/During/?
Quoted 26 lines
xen/arch/arm/smpboot.c +12
-1+* CPU dts node doesn't have numa-node-id info, the NUMA will be +* treated as off, all CPU will be set to a FAKE node 0. So if we +* get numa-node-id failed here, we should set nid to 0. +*/ + nid = 0; + } +#endif +/* * 8 MSBs must be set to 0 in the DT since the reg property * defines the MPIDR[23:0] @@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void) {printk ("cpu%d init failed (hwid %" PRIregister"): %d\n" , i, hwid, rc); tmp_map[i] =MPIDR_INVALID ; + node_map[i] =NUMA_NO_NODE ; } - else + else { tmp_map[i] = hwid; + node_map[i] = nid; + } }if ( !bootcpu_valid )
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 283
Quoted 7 lines
xen/arch/arm/smpboot.c +1
-0@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)continue ;cpumask_set_cpu (i, &cpu_possible_map);cpu_logical_map (i) = tmp_map[i]; +numa_set_node (i, node_map[i]); } } >
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info
In reply to Wei ChenHi Wei,
Quoted 14 lines · Wei Chen
These two helpers are architecture APIs that are required by
nodes_cover_memory.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)xen/arch/arm/numa.c +0
-0@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end));
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 136
Quoted 10 lines
xen/arch/arm/numa.c +8
-0return ; } + +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return bootinfo.mem.nr_banks; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end)
They are physical address, so we should use "paddr_t" as on system such as 32-bit Arm, "unsigned long" is not enough to cover all the physical address.
As you change the type, I would also suggest to change the bank from an int to an unsigned int.
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 142
Quoted 7 lines
xen/arch/arm/numa.c +6
-0+{ + *start = bootinfo.mem.bank[bank].start; + *end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size; + +return 0 ; +}
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common
In reply to Wei ChenHi Wei,
Quoted 3 lines · Wei Chen
Both
memory_nodes_parsedandprocessor_nodes_parsedare using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.
Looking at the usage, they both call:
numa_set...(..., bitmap)
So rather than exporting the two helpers, could we simply add helpers to abstract it?
Quoted 14 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 2 -- xen/arch/x86/srat.c | 3 --- xen/common/numa.c | 3 +++ xen/include/xen/nodemask.h | 2 ++ 4 files changed, 5 insertions(+), 5 deletions(-)xen/arch/arm/numa_device_tree.c +0
-0
xen/arch/arm/numa_device_tree.c line 29
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +0
-2@@ -25,8 +25,6 @@#include <asm/setup.h> s8 device_tree_numa =0 ; -static nodemask_t processor_nodes_parsed__initdata ; -static nodemask_t memory_nodes_parsed__initdata ;
This is code that was introduced in a previous patch. In general, it is better to do the rework first and then add the new code. This makes easier to follow series as the code added is not changed.
Quoted 37 lines
xen/arch/arm/numa_device_tree.c +0
-0static int srat_disabled (void ) {xen/arch/x86/srat.c +0
-3@@ -24,9 +24,6 @@static struct acpi_table_slit *__read_mostly acpi_slit; -static nodemask_t memory_nodes_parsed__initdata ; -static nodemask_t processor_nodes_parsed__initdata ; -struct pxm2node {unsigned pxm;nodeid_t node;xen/common/numa.c +3
-0@@ -35,6 +35,9 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +nodemask_t memory_nodes_parsed__initdata ; +nodemask_t processor_nodes_parsed__initdata ; +bool numa_off;/* xen/include/xen/nodemask.h +0
-0
xen/include/xen/nodemask.h static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits) · line 283
Quoted 8 lines
xen/include/xen/nodemask.h +2
-0@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits)*/ extern nodemask_t node_online_map; +extern nodemask_t memory_nodes_parsed; +extern nodemask_t processor_nodes_parsed;#if MAX_NUMNODES >1 #define num_online_nodes () nodes_weight(node_online_map)
Cheers,
Signature · 1 line
Julien Grall
Thanks for the big contribution!
I just wanted to let you know that the series passed all the gitlab-ci build tests without issues.
The runtime tests originally failed due to unrelated problems (there was a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying issue and restarted the failed tests and now they passed.
This is the pipeline:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940
There are still two runtime x86 tests that fail but I don't think the failures are related to your series.
Quoted 120 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto extern
xen: movenuma_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to commontools/libs/util/libxlu_pci.c | 3 +- xen/arch/arm/Kconfig | 10 + xen/arch/arm/Makefile | 2 + xen/arch/arm/arm64/head.S | 9 +- xen/arch/arm/bootfdt.c | 8 +- xen/arch/arm/domain_build.c | 17 +- xen/arch/arm/efi/efi-boot.h | 25 -- xen/arch/arm/numa.c | 162 +++++++++ xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++ xen/arch/arm/platform.c | 4 +- xen/arch/arm/setup.c | 14 + xen/arch/arm/smpboot.c | 37 +- xen/arch/x86/Kconfig | 2 +- xen/arch/x86/numa.c | 421 +---------------------- xen/arch/x86/srat.c | 147 +------- xen/common/Kconfig | 3 + xen/common/Makefile | 1 + xen/common/libfdt/fdt_ro.c | 15 + xen/common/numa.c | 588 ++++++++++++++++++++++++++++++++ xen/common/page_alloc.c | 2 +- xen/drivers/acpi/Kconfig | 3 +- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-arm/numa.h | 33 ++ xen/include/asm-arm/setup.h | 6 + xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/config.h | 1 - xen/include/asm-x86/numa.h | 65 +--- xen/include/asm-x86/setup.h | 1 - xen/include/xen/libfdt/libfdt.h | 25 ++ xen/include/xen/nodemask.h | 2 + xen/include/xen/numa.h | 80 +++++ 31 files changed, 1325 insertions(+), 659 deletions(-) create mode 100644 xen/arch/arm/numa.c create mode 100644 xen/arch/arm/numa_device_tree.c create mode 100644 xen/common/numa.cSignature · 1 line
2.25.1
RE: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default
In reply to Jan BeulichHi Jan, Julien,
Quoted 29 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月25日 21:36
To: Julien Grall <[email protected]>; Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by defaultQuoted 7 lines · Julien Grall
Quoted 6 lines · Wei Chen
xen/include/asm-arm/numa.h +0
-0@@ -5,7 +5,15 @@typedef u8 nodeid_t ;Quoted 11 lines
xen/include/asm-arm/numa.h line 13
Quoted 7 lines
xen/include/asm-arm/numa.h +6
-1-#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/ Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to
xen/numa.h.
Quoted 5 lines
To be honest - if this gets moved, please at least consider making it a proper
Kconfigsetting. Just as much a the number of CPUs can be configured, the number of nodes should be possible to choose by the build manager. Of course - if it's not too much trouble ...
Ok
Quoted 1 line
Jan
RE: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
In reply to Julien GrallHi Julien,
Quoted 51 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 26 lines
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmHi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.Quoted 6 lines
IHMO
numa_set_node()should behave exactly the same way on x86 and Arm because this is going to be used by the common code.From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?Quoted 8 lines
Yes, in patch#27, in
smpboot.c,dt_smp_init_cpusfunction. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invokenuma_set_nodeto create CPU-NODE map. But in our testing, we found when NUMA init failed,numa_set_nodestill can set CPU to a offline or invalid NODE. So we're usingnode_online_mapto prevent this behavior. Otherwise we have to checknode_online_mapeverywhere before we callnuma_set_node.
Quoted 1 line
What do you mean by invalid NODE? Is it
0xFF(NUMA_NO_NODE)?
No, I mean some wrong content in device tree. For example, if the dtb set a wrong numa-node-id in CPU dt-node.
Quoted 20 lines
Quoted 4 lines
x86 actually is doing the same way, but it handles
node_online_mapcheck out ofnuma_set_node:Right...
Quoted 5 lines
Quoted 2 lines
I think
numa_set_node()will want to be implemented in common code.See my above comment. If x86 is ok, I think yes, we can do it in common code.
... on x86, this check is performed outside of
numa_set_node()for one caller whereas on Arm you are adding it innuma_set_node().For example,
numa_set_node()can be called withNUMA_NO_NODE. On x86, we would set cpu_to_node[] to that value. However, if I am not mistaken, on Arm we would set the value to 0.
Quoted 9 lines
This will change the behavior of users to
cpu_to_node()later on (such asXEN_SYSCTL_cputopoinfo).NUMA is not something architecture specific, so I dont't think the implementation should differ here.
In this case, I think
numa_set_node()shouldn't check if the node is valid. Instead, the caller should take care of it if it is important.
Yes, I agree. I will change it in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type
In reply to Julien GrallHi Julien,
Quoted 18 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:39
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node typeHi Wei,
Quoted 5 lines · Wei Chen
In later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce
fdt_node_check_typein this patch to avoid redundant code in subsequent patches.
Quoted 9 lines
Quoted 4 lines
Signed-off-by Wei Chen <[email protected]>--- xen/common/libfdt/fdt_ro.c | 15 +++++++++++++++ xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++This is meant to be a verbatim copy of libfdt. So I am not entirely in favor of adding a new function therefore without been upstreamed to libfdt first.
Oh, if we need to upstream this change in libfdt. I think I'd better to remove this change in libfdt. Because we can implement type checking in other place, and I don't want to introduce a dependency on external repo upstream in this series.
Quoted 11 lines
xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int
Quoted 7 lines
2 files changed, 40 insertions(+)xen/common/libfdt/fdt_ro.c +0
-0@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, intnodeoffset,
Quoted 3 lines
xen/common/libfdt/fdt_ro.c +0
-0return 1 ; }
Quoted 15 lines
xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int nodeoffset, · line 557
Quoted 10 lines
xen/common/libfdt/fdt_ro.c +10
-0+int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type) +{ +const void *prop; +int len; + + prop =fdt_getprop (fdt, nodeoffset,"device_type" , &len); +if (!prop) +return len; +if (fdt_stringlist_contains (prop, len, type))The "
device_type" is not a list of string. So I am a bit confused why you are using this helper. Shouldn't we simply check that the property value and type matches?
Yes, I think you're right. This function was based on the modification of fdt_node_check_compatible, and I forgot to replace fdt_stringlist_contains. And, as I reply above, we can simply the check. And I will implement it out of libfdt.
Quoted 56 lines
Review
Quoted 9 lines
xen/common/libfdt/fdt_ro.c +5
-0+return 0 ; + else + return 1; +} +int fdt_node_offset_by_compatible (const void *fdt,int startoffset,const char *compatible) {(quoted diff) +0
-0
b/xen/include/xen/libfdt/libfdt.hxen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt,
Quoted 4 lines
index
7c75688a39..7e4930dbcd100644xen/include/xen/libfdt/libfdt.h +0
-0@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt,
uint32_tphandle);xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
Quoted 11 lines
xen/include/xen/libfdt/libfdt.h +8
-0int fdt_node_check_compatible (const void *fdt,int nodeoffset,const char *compatible); +/** +* fdt_node_check_type: check a node's device_type property +* @fdt: pointer to the device tree blob +* @nodeoffset: offset of a tree node +* @type: string to match against +* +* +* fdt_node_check_type() returns 0 if the given node contains a '
device_type'xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, · line 810
Quoted 1 line
xen/include/xen/libfdt/libfdt.h +1
-0+* property with the given string as one of its elements, it returns non-zero
xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
Quoted 8 lines
xen/include/xen/libfdt/libfdt.h +8
-0+* otherwise, or on error. +* +* returns: +* 0, if the node has a 'device_type' property listing the given string +* 1, if the node has a 'device_type' property, but it does not list +* the given string +* -FDT_ERR_NOTFOUND, if the given node has no 'device_type' property +* -FDT_ERR_BADOFFSET, if nodeoffset does not refer to a
BEGIN_NODEtagxen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
Quoted 10 lines
xen/include/xen/libfdt/libfdt.h +8
-0+* -FDT_ERR_BADMAGIC, +* -FDT_ERR_BADVERSION, +* -FDT_ERR_BADSTATE, +* -FDT_ERR_BADSTRUCTURE, standard meanings +*/ +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type); +/** * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value
xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle); · line 829
Quoted 2 lines
xen/include/xen/libfdt/libfdt.h +0
-0* @fdt: pointer to the device tree blob Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
In reply to Julien GrallHi Julien,
Quoted 48 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Wei,
Quoted 7 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-0return 0 ; }
Quoted 7 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?
In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.
But, yes, maybe we can abstract some common code to functions, that can be reused in two functions. I would try it in next version.
Quoted 26 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +7
-0+struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; +xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 86
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) {
Quoted 8 lines
Xen coding style is using:
if ( ... )
{Note that I may not comment on all the occurents, so please check the other places.
Ok.
Quoted 53 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7
-0+/* +* Overlaps with other memblk in the same node, warning here. +* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const
Quoted 43 lines
xen/arch/arm/numa_device_tree.c +39
-0+ node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++; + +return 0 ; +} +/* Parse CPU NUMA node info */ int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) {@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(constvoid *fdt, int node)
Quoted 1 line
Quoted 11 lines
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 152
Quoted 7 lines
return dtb_numa_processor_affinity_init(nid);}
xen/arch/arm/numa_device_tree.c +5
-0+ +/* Parse memory node NUMA info */ +int __init +device_tree_parse_numa_memory_node (const void *fdt,int node, +const char *name,uint32_t addr_cells,uint32_t size_cells)This is pretty much a copy of
process_memory_node(). Can we consider to collect the NUMA ID from there? If not, can we at least abstract the code?
I had tried to parse NUMA id in process_memory_node or in early_scan_node. But I found, this change would make NUMA init code in different places. And I prefer a unify NUMA init entry. Because it may be good for maintenance. When we want to disable NUMA, we just need to disable it in one place. But you're right, we still can do some code abstraction. I will do it in next version.
Quoted 38 lines
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12
-0+{ +uint32_t nid; +int ret =0 , len; +paddr_t addr, size; +const struct fdt_property *prop; +uint32_t idx, ranges; +const __be32 *addresses; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+return -EINVAL ; + } + + prop =fdt_get_property (fdt, node,"reg" , &len); +if ( !prop ) + {xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+printk (XENLOG_WARNING +"fdt: node `%s': missing `reg' property\n" , name); +return -EINVAL ; + } + + addresses = (const __be32 *)prop->data; + ranges = len / (sizeof (__be32)* (addr_cells + size_cells)); +for ( idx =0 ; idx < ranges; idx++ ) + { + device_tree_get_reg(&addresses, addr_cells, size_cells, &addr,&size);
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+/* Skip zero size ranges */ +if ( !size ) +continue ; + + ret =dtb_numa_memory_affinity_init (nid, addr, size); + if ( ret ) { + printk(XENLOG_WARNING + "NUMA: process range#%d addr = %lx size=%lx
Quoted 5 lines
failed!\n",
s/%d/%u/ as idx is an unsigned int
s/%lx/%"PRI_paddr"/ as addr and size arepaddr_t.
OK
Quoted 21 lines
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +9
-0+ idx, addr, size); +return -EINVAL ; + } + } + +if ( idx ==0 ) + { + printk(XENLOG_ERR + "bad property in memory node, idx=%d ret=%d\n", idx,ret);
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 131
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return 0 ; +}Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
In reply to Julien GrallHi Julien,
Quoted 10 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:56
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Quoted 7 lines
Hi Wei,
Quoted 2 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a
s/introduce/introduces/
OK
Quoted 16 lines
Review
Quoted 9 lines
new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt,
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt,int node,
Quoted 1 line
Quoted 14 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 210
Quoted 10 lines
return 0;}
xen/arch/arm/numa_device_tree.c +8
-0+ +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;
entry_countand i should be unsigned. len unfortunately can't becausefdt_get_propertyexpects a signed int.
OK
Quoted 17 lines
Quoted 17 lines
xen/arch/arm/numa_device_tree.c +17
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + } + +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ;
Quoted 9 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 231
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+ } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 )I understand that
entry_countmay be 0. But I can't see how it can be negative as the property len cannot be (even if it is a signed type). So I think this wants to be "== 0".
From the fdt_get_property's comment, when prop is NULL, the len can be negative. But, yes, prop==NULL check will handled before this code. negative len will not reach here. I am ok to change it to "== 0"
Quoted 7 lines
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7
-0+ { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data;
Quoted 9 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 244
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;You can use
dt_next_cell()which will update the pointer for you.
Thanks
Quoted 33 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 269
Quoted 11 lines
xen/arch/arm/numa_device_tree.c +10
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance); + } + +return 0 ; +}Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
In reply to Julien GrallHi Julien,
Quoted 52 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA systemHi Wei,
xen/arch/arm/setup.c void __init start_xen(unsigned long
Quoted 19 lines · Wei Chen
When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/setup.c | 6 ++++++ xen/arch/arm/smpboot.c | 6 ++++++ xen/include/asm-arm/numa.h | 1 + 3 files changed, 13 insertions(+)xen/arch/arm/setup.c +0
-0@@ -918,6 +918,12 @@ void __init start_xen(unsigned long
boot_phys_offset,Quoted 20 lines
xen/arch/arm/setup.c +6
-0processor_id (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (0 ); +smp_init_cpus (); cpus =smp_get_max_cpus ();printk (XENLOG_INFO "SMP: Allowing %u CPUs\n" , cpus);xen/arch/arm/smpboot.c +0
-0@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb ();
Quoted 10 lines
xen/arch/arm/smpboot.c void start_secondary(void) · line 366
Quoted 6 lines
xen/arch/arm/smpboot.c +6
-0+/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +On x86,
numa_add_cpu()will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?
Of course we can invoke numa_add_cpu before cpu_up as x86. But in my tests, I found when cpu bring up failed, this cpu still be add to NUMA. Although this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.
Quoted 11 lines
Quoted 11 lines
xen/arch/arm/smpboot.c +0
-0/* Now report this CPU is up */ cpumask_set_cpu (cpuid, &cpu_online_map);xen/include/asm-arm/numa.h +0
-0@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;#define __node_distance (a, b) (20)
Quoted 6 lines
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 62
Quoted 2 lines
xen/include/asm-arm/numa.h +1
-0#define numa_init (x) do { } while (0) +#define numa_add_cpu (x) do { } while (0)This is a stubs for a common helper. So I think this wants to be moved in the !CONFIG_NUMA in
xen/numa.h.
OK
Quoted 9 lines
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 65
Quoted 4 lines
xen/include/asm-arm/numa.h +0
-0#define numa_set_node (x, y) do { } while (0)#endif Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map
In reply to Julien GrallHi Julien,
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:07
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creatingcpu_logical_mapHi Wei,
Quoted 3 lines · Wei Chen
Sometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.
Quoted 9 lines
This commit message gives the impression that you are trying to fix a bug. However, what you are explaining is the reason why the code will use the logical ID rather than physical ID.
I think the commit message should explain what the patch is doing. You can then add an explanation why you are using the CPU logical ID. Something like "Note we storing the CPU logical ID because...".
Ok
Quoted 35 lines
Quoted 24 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)xen/arch/arm/smpboot.c +5
-0@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void) { [0 ... NR_CPUS - 1] = MPIDR_INVALID }; +static nodeid_t node_map[NR_CPUS ]__initdata = + { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE + };bool bootcpu_valid = false; +uint32_t nid =0 ;int rc; mpidr = boot_cpu_data.mpidr.bits &MPIDR_HWID_MASK ;xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 182
Quoted 6 lines
xen/arch/arm/smpboot.c +3
-0@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)continue ; } +#ifdef CONFIG_DEVICE_TREE_NUMA +/* +* When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 185
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+* from CPU dts node, otherwise the nid is always 0. +*/ +if ( !dt_property_read_u32 (cpu,"numa-node-id" , &nid) )
Quoted 7 lines
You can avoid the #ifdef by writing:
if (
IS_ENABLED(CONFIG_DEVICE_TREE_NUMA)&& ... )However, I would using
CONFIG_NUMAbecause this code is already DT specific. So we can shorten the name a bit.
OK
Quoted 5 lines
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 188
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+ { + printk(XENLOG_WARNING + "cpu[%d] dts path: %s: doesn't have numa infomation!\n",s/information/information/
OK
Quoted 7 lines
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void)
Quoted 4 lines
xen/arch/arm/smpboot.c +3
-0+ cpuidx, dt_node_full_name(cpu)); +/* +* The the early stage of NUMA initialization, when Xen found any
s/The/During/?
Oh, yes, I will fix it.
Quoted 42 lines
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 192
Quoted 2 lines
xen/arch/arm/smpboot.c +1
-0+* CPU dts node doesn't have numa-node-id info, the NUMA will be
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 193
Quoted 1 line
xen/arch/arm/smpboot.c +1
-0+* treated as off, all CPU will be set to a FAKE node 0. So if we
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 257
Quoted 12 lines
xen/arch/arm/smpboot.c +6
-0+* get numa-node-id failed here, we should set nid to 0. +*/ + nid = 0; + } +#endif +/* * 8 MSBs must be set to 0 in the DT since the reg property * defines the MPIDR[23:0] @@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void) { printk("cpu%d init failed (hwid %"PRIregister"): %d\n", i,hwid, rc);
Quoted 12 lines
xen/arch/arm/smpboot.c +4
-1tmp_map[i] =MPIDR_INVALID ; + node_map[i] =NUMA_NO_NODE ; } -else +else { tmp_map[i] = hwid; + node_map[i] = nid; + } }if ( !bootcpu_valid )xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 283
Quoted 7 lines
xen/arch/arm/smpboot.c +1
-0@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)continue ;cpumask_set_cpu (i, &cpu_possible_map);cpu_logical_map (i) = tmp_map[i]; +numa_set_node (i, node_map[i]); } } >Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info
In reply to Julien GrallHi Julien,
Quoted 41 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:10
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map infoHi Wei,
Quoted 14 lines · Wei Chen
These two helpers are architecture APIs that are required by
nodes_cover_memory.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)xen/arch/arm/numa.c +0
-0@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end));xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 136
Quoted 10 lines
xen/arch/arm/numa.c +8
-0return ; } + +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return bootinfo.mem.nr_banks; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end)They are physical address, so we should use "
paddr_t" as on system such as 32-bit Arm, "unsigned long" is not enough to cover all the physical address.
Quoted 3 lines
As you change the type, I would also suggest to change the bank from an int to an unsigned int.
I will fix them in next version.
Quoted 12 lines
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 142
Quoted 7 lines
xen/arch/arm/numa.c +6
-0+{ + *start = bootinfo.mem.bank[bank].start; + *end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size; + +return 0 ; +}Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common
In reply to Julien GrallHi Juilien,
Quoted 16 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:17
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to commonHi Wei,
Quoted 3 lines · Wei Chen
Both
memory_nodes_parsedandprocessor_nodes_parsedare using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.
Quoted 7 lines
Looking at the usage, they both call:
numa_set...(..., bitmap)
So rather than exporting the two helpers, could we simply add helpers to abstract it?
I will try to fix it in next version.
Quoted 18 lines
Review
Quoted 11 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 2 -- xen/arch/x86/srat.c | 3 --- xen/common/numa.c | 3 +++ xen/include/xen/nodemask.h | 2 ++ 4 files changed, 5 insertions(+), 5 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 6 lines
index
27ffb72f7b..f74b7f6427100644xen/arch/arm/numa_device_tree.c +0
-0@@ -25,8 +25,6 @@#include <asm/setup.h>
Quoted 8 lines
xen/arch/arm/numa_device_tree.c line 29
Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-2s8 device_tree_numa =0 ; -static nodemask_t processor_nodes_parsed__initdata ; -static nodemask_t memory_nodes_parsed__initdata ;This is code that was introduced in a previous patch. In general, it is better to do the rework first and then add the new code. This makes easier to follow series as the code added is not changed.
Yes, I will fix it in next version.
Quoted 51 lines
xen/include/xen/nodemask.h static inline int __cycle_node(int n, const
Quoted 36 lines
xen/arch/arm/numa_device_tree.c +0
-0static int srat_disabled (void ) {xen/arch/x86/srat.c +0
-3@@ -24,9 +24,6 @@static struct acpi_table_slit *__read_mostly acpi_slit; -static nodemask_t memory_nodes_parsed__initdata ; -static nodemask_t processor_nodes_parsed__initdata ; -struct pxm2node {unsigned pxm;nodeid_t node;xen/common/numa.c +3
-0@@ -35,6 +35,9 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +nodemask_t memory_nodes_parsed__initdata ; +nodemask_t processor_nodes_parsed__initdata ; +bool numa_off;/* xen/include/xen/nodemask.h +0
-0@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const
nodemask_t*maskp, int nbits)xen/include/xen/nodemask.h static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits) · line 283
Quoted 9 lines
xen/include/xen/nodemask.h +2
-0*/ extern nodemask_t node_online_map; +extern nodemask_t memory_nodes_parsed; +extern nodemask_t processor_nodes_parsed;#if MAX_NUMNODES >1 #define num_online_nodes () nodes_weight(node_online_map)Cheers,
--
Julien Grall
Hi Stefano,
Quoted 19 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月26日 8:09
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]
Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64Thanks for the big contribution!
I just wanted to let you know that the series passed all the gitlab-ci build tests without issues.
The runtime tests originally failed due to unrelated problems (there was a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying issue and restarted the failed tests and now they passed.
Quoted 7 lines
This is the pipeline:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940There are still two runtime x86 tests that fail but I don't think the failures are related to your series.
Thanks for testing this series : )
Quoted 121 lines · Wei Chen
Quoted 120 lines
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto extern
xen: movenuma_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to commontools/libs/util/libxlu_pci.c | 3 +- xen/arch/arm/Kconfig | 10 + xen/arch/arm/Makefile | 2 + xen/arch/arm/arm64/head.S | 9 +- xen/arch/arm/bootfdt.c | 8 +- xen/arch/arm/domain_build.c | 17 +- xen/arch/arm/efi/efi-boot.h | 25 -- xen/arch/arm/numa.c | 162 +++++++++ xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++ xen/arch/arm/platform.c | 4 +- xen/arch/arm/setup.c | 14 + xen/arch/arm/smpboot.c | 37 +- xen/arch/x86/Kconfig | 2 +- xen/arch/x86/numa.c | 421 +---------------------- xen/arch/x86/srat.c | 147 +------- xen/common/Kconfig | 3 + xen/common/Makefile | 1 + xen/common/libfdt/fdt_ro.c | 15 + xen/common/numa.c | 588 ++++++++++++++++++++++++++++++++ xen/common/page_alloc.c | 2 +- xen/drivers/acpi/Kconfig | 3 +- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-arm/numa.h | 33 ++ xen/include/asm-arm/setup.h | 6 + xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/config.h | 1 - xen/include/asm-x86/numa.h | 65 +--- xen/include/asm-x86/setup.h | 1 - xen/include/xen/libfdt/libfdt.h | 25 ++ xen/include/xen/nodemask.h | 2 + xen/include/xen/numa.h | 80 +++++ 31 files changed, 1325 insertions(+), 659 deletions(-) create mode 100644 xen/arch/arm/numa.c create mode 100644 xen/arch/arm/numa_device_tree.c create mode 100644 xen/common/numa.c--
2.25.1
Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
In reply to Wei ChenQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 56 lines
Quoted 48 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Wei,
Quoted 7 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-0return 0 ; }Quoted 7 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?
Quoted 4 lines
In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.
It think you can get away from distinguishing between ACPI and DT in that helper:
* ma->flags & ACPI_SRAT_MEM_HOTPLUGGABLE could be replace by an
argument indicating whether the region is hotpluggable (this would always be false for DT)
* Access to memblk_hotplug can be stubbed (in the future we may want
to consider memory hotplug even on Arm).
Do you still have the "if else" version? If so can you post it?
Cheers,
Signature · 1 line
Julien Grall