Skip to content

RFC Add device tree based NUMA support to Arm64

Wei Chen xen-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest40 patches 155 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/40tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdftools/libs/util/libxlu_pci.c+1 -22 replies
2/40xen/arm: Print a 64-bit number in hex from early uartxen/arch/arm/arm64/head.S+5 -42 replies
3/40xen/x86: Initialize memnodemapsize while faking NUMA nodexen/arch/x86/numa.c+2 -03 replies
4/40xen/arm: return default DMA bit width when platform is not setxen/arch/arm/platform.c xen/common/page_alloc.c+4 -29 replies
5/40xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0xen/arch/arm/domain_build.c+10 -12 replies
6/40xen: decouple NUMA from ACPI in Kconfigxen/arch/x86/Kconfig xen/common/Kconfig xen/drivers/acpi/Kconfig +2 more+7 -44 replies
7/40xen/arm: use !CONFIG_NUMA to keep fake NUMA APIxen/include/asm-arm/numa.h+4 -07 replies
8/40xen/x86: Move NUMA memory node map functions to commonxen/arch/x86/numa.c xen/common/Makefile xen/common/numa.c +2 more+167 -1432 replies
9/40xen/x86: Move numa_add_cpu_node to commonxen/arch/x86/numa.c xen/common/numa.c xen/include/asm-x86/numa.h +1 more+13 -102 replies
10/40xen/x86: Move NR_NODE_MEMBLKS macro to commonxen/include/asm-x86/acpi.h xen/include/xen/numa.h+1 -11 reply
11/40xen/x86: Move NUMA nodes and memory block ranges to commonxen/arch/x86/srat.c xen/common/numa.c xen/include/asm-x86/numa.h +1 more+66 -582 replies
12/40xen/x86: Move numa_initmem_init to commonxen/arch/x86/numa.c xen/common/numa.c xen/include/asm-x86/numa.h +2 more+128 -1263 replies
13/40xen/arm: introduce numa_set_node for Armxen/arch/arm/Makefile xen/arch/arm/numa.c xen/include/asm-arm/numa.h +2 more+35 -14 replies
14/40xen/arm: set NUMA nodes max number to 64 by defaultxen/include/asm-arm/numa.h+9 -13 replies
15/40xen/x86: move NUMA API from x86 header to common headerxen/include/asm-x86/numa.h xen/include/xen/numa.h+13 -130 replies
16/40xen/arm: Create a fake NUMA node to use common codexen/arch/arm/numa.c xen/arch/arm/setup.c xen/include/asm-arm/numa.h+72 -04 replies
17/40xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64xen/arch/arm/Kconfig+10 -06 replies
18/40xen/arm: Keep memory nodes in dtb for NUMA when boot from EFIxen/arch/arm/bootfdt.c xen/arch/arm/efi/efi-boot.h xen/include/asm-arm/setup.h+13 -266 replies
19/40xen: fdt: Introduce a helper to check fdt node typexen/common/libfdt/fdt_ro.c xen/include/xen/libfdt/libfdt.h+40 -02 replies
20/40xen/arm: implement node distance helpers for Arm64xen/arch/arm/numa.c xen/include/asm-arm/numa.h xen/include/asm-x86/numa.h +1 more+56 -32 replies
21/40xen/arm: introduce device_tree_numa as a switch for device tree NUMAxen/arch/arm/Makefile xen/arch/arm/numa_device_tree.c xen/include/asm-arm/numa.h+38 -02 replies
22/40xen/arm: introduce a helper to parse device tree processor nodexen/arch/arm/numa_device_tree.c+39 -212 replies
23/40xen/arm: introduce a helper to parse device tree memory nodexen/arch/arm/numa_device_tree.c+130 -011 replies
24/40xen/arm: introduce a helper to parse device tree NUMA distance mapxen/arch/arm/numa_device_tree.c+67 -011 replies
25/40xen/arm: unified entry to parse all NUMA data from device treexen/arch/arm/numa_device_tree.c xen/include/asm-arm/numa.h+29 -34 replies
26/40xen/arm: Add boot and secondary CPU to NUMA systemxen/arch/arm/setup.c xen/arch/arm/smpboot.c xen/include/asm-arm/numa.h+13 -06 replies
27/40xen/arm: build CPU NUMA node map while creating cpu_logical_mapxen/arch/arm/smpboot.c+30 -12 replies
28/40xen/x86: decouple nodes_cover_memory with E820 mapxen/arch/x86/numa.c xen/arch/x86/srat.c xen/include/xen/numa.h+25 -52 replies
29/40xen/arm: implement Arm arch helpers Arm to get memory map infoxen/arch/arm/numa.c+14 -02 replies
30/40xen: move NUMA memory and CPU parsed nodemasks to commonxen/arch/arm/numa_device_tree.c xen/arch/x86/srat.c xen/common/numa.c +1 more+5 -52 replies
31/40xen/x86: move nodes_cover_memory to commonxen/arch/x86/srat.c xen/common/numa.c xen/include/xen/numa.h+41 -402 replies
32/40xen/x86: make acpi_scan_nodes to be neutralxen/arch/x86/srat.c xen/common/numa.c xen/include/asm-x86/acpi.h+5 -32 replies
33/40xen: export bad_srat and srat_disabled to externxen/arch/arm/numa_device_tree.c xen/arch/x86/srat.c xen/include/asm-x86/numa.h +1 more+5 -40 replies
34/40xen: move numa_scan_nodes from x86 to commonxen/arch/x86/srat.c xen/common/numa.c xen/include/asm-x86/acpi.h +1 more+55 -574 replies
35/40xen: enable numa_scan_nodes for device tree based NUMAxen/common/numa.c+1 -12 replies
36/40xen/arm: keep guest still be NUMA unwarexen/arch/arm/domain_build.c+6 -02 replies
37/40xen: introduce an arch helper to do NUMA init failed fallbackxen/arch/arm/numa.c xen/common/numa.c xen/include/asm-arm/numa.h +1 more+23 -05 replies
38/40xen/arm: enable device tree based NUMA in system initxen/arch/arm/numa.c xen/include/asm-arm/numa.h+26 -266 replies
39/40xen/x86: move numa_setup to common to support NUMA switch in command linexen/arch/x86/numa.c xen/common/numa.c xen/include/xen/numa.h+34 -364 replies
40/40xen/x86: move dump_numa info hotkey to commonxen/arch/x86/numa.c xen/common/numa.c+144 -1420 replies

Reviews trailers per patch

3 of 40 patches have a Reviewed-by or Acked-by.

PatchJan BeulichJulien GrallMatt FlemingSteve Capper
1/40
2/40A-b
3/40
4/40
5/40
6/40R-b
7/40
8/40
9/40
10/40
11/40
12/40
13/40
14/40
15/40
16/40
17/40
18/40R-b A-b
19/40
20/40
21/40
22/40
23/40
24/40
25/40
26/40
27/40
28/40
29/40
30/40
31/40
32/40
33/40
34/40
35/40
36/40
37/40
38/40
39/40
40/40

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v1 thread

Cover letter [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.

Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.

So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.

Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.

So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.

Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern
xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common

 tools/libs/util/libxlu_pci.c    |   3 +-
 xen/arch/arm/Kconfig            |  10 +
 xen/arch/arm/Makefile           |   2 +
 xen/arch/arm/arm64/head.S       |   9 +-
 xen/arch/arm/bootfdt.c          |   8 +-
 xen/arch/arm/domain_build.c     |  17 +-
 xen/arch/arm/efi/efi-boot.h     |  25 --
 xen/arch/arm/numa.c             | 162 +++++++++
 xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
 xen/arch/arm/platform.c         |   4 +-
 xen/arch/arm/setup.c            |  14 +
 xen/arch/arm/smpboot.c          |  37 +-
 xen/arch/x86/Kconfig            |   2 +-
 xen/arch/x86/numa.c             | 421 +----------------------
 xen/arch/x86/srat.c             | 147 +-------
 xen/common/Kconfig              |   3 +
 xen/common/Makefile             |   1 +
 xen/common/libfdt/fdt_ro.c      |  15 +
 xen/common/numa.c               | 588 ++++++++++++++++++++++++++++++++
 xen/common/page_alloc.c         |   2 +-
 xen/drivers/acpi/Kconfig        |   3 +-
 xen/drivers/acpi/Makefile       |   2 +-
 xen/include/asm-arm/numa.h      |  33 ++
 xen/include/asm-arm/setup.h     |   6 +
 xen/include/asm-x86/acpi.h      |   4 -
 xen/include/asm-x86/config.h    |   1 -
 xen/include/asm-x86/numa.h      |  65 +---
 xen/include/asm-x86/setup.h     |   1 -
 xen/include/xen/libfdt/libfdt.h |  25 ++
 xen/include/xen/nodemask.h      |   2 +
 xen/include/xen/numa.h          |  80 +++++
 31 files changed, 1325 insertions(+), 659 deletions(-)
 create mode 100644 xen/arch/arm/numa.c
 create mode 100644 xen/arch/arm/numa_device_tree.c
 create mode 100644 xen/common/numa.c
Signature · 1 line
2.25.1