Skip to content

RFC PMEM NUMA node and hotness accounting/migration

Fengguang Wu kvm linux-mm lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v2 RFC Latest21 patches 41 repliesIn review

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/21e820: cheat PMEM as DRAMarch/x86/kernel/e820.c+2 -15 replies
2/21acpi/numa: memorize NUMA node type from SRAT tablearch/x86/include/asm/numa.h arch/x86/mm/numa.c drivers/acpi/numa.c+9 -00 replies
3/21x86/numa_emulation: fix fake NUMA in uniform casearch/x86/mm/numa_emulation.c+15 -10 replies
4/21x86/numa_emulation: pass numa node type to fake nodesarch/x86/mm/numa_emulation.c+14 -00 replies
5/21mmzone: new pgdat flags for DRAM and PMEMinclude/linux/mmzone.h+26 -00 replies
6/21x86,numa: update numa node typearch/x86/mm/numa.c+1 -00 replies
7/21mm: export node type {pmem|dram} under /sys/bus/nodedrivers/base/node.c+10 -00 replies
8/21mm: introduce and export pgdat peer_nodedrivers/base/node.c include/linux/mmzone.h mm/page_alloc.c+52 -02 replies
9/21mm: avoid duplicate peer target nodemm/page_alloc.c+6 -00 replies
10/21mm: build separate zonelist for PMEM and DRAM nodemm/mempolicy.c mm/page_alloc.c+43 -133 replies
11/21kvm: allocate page table pages from DRAMarch/x86/kvm/mmu.c+11 -14 replies
12/21x86/pgtable: allocate page table pages from DRAMarch/x86/include/asm/pgalloc.h arch/x86/mm/pgtable.c+25 -70 replies
13/21x86/pgtable: dont check PMD accessed bitarch/x86/include/asm/pgtable.h+2 -10 replies
14/21kvm: register in mm_structinclude/linux/mm_types.h virt/kvm/kvm_main.c+14 -03 replies
15/21ept-idle: EPT walk for virtual machinearch/x86/kvm/ept_idle.c arch/x86/kvm/ept_idle.h+753 -00 replies
16/21mm-idle: mm_walk for normal taskarch/x86/kvm/ept_idle.c mm/pagewalk.c+205 -00 replies
17/21proc: introduce /proc/PID/idle_pagesfs/proc/base.c fs/proc/internal.h fs/proc/task_mmu.c+57 -00 replies
18/21kvm-ept-idle: enable modulearch/x86/kvm/Kconfig arch/x86/kvm/Makefile+15 -00 replies
19/21mm/migrate.c: add move_pages(MPOL_MF_SW_YOUNG) flagmm/migrate.c+10 -30 replies
20/21mm/vmscan.c: migrate anon DRAM pages to PMEM nodemm/vmscan.c+33 -00 replies
21/21mm/vmscan.c: shrink anon list if can migrate to PMEMmm/vmscan.c+3 -20 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [RFC][PATCH v2 00/21] PMEM NUMA node and hotness accounting/migration

This is an attempt to use NVDIMM/PMEM as volatile NUMA memory that's transparent to normal applications and virtual machines.

The code is still in active development. It's provided for early design review.

Key functionalities:

1) create and describe PMEM NUMA node for NVDIMM memory
2) dumb /proc/PID/idle_pages interface, for user space driven hot page accounting
3) passive kernel cold page migration in page reclaim path
4) improved move_pages() for active user space hot/cold page migration

(1) is foundation for transparent usage of NVDIMM for normal apps and virtual machines. (2-4) enable auto placing hot pages in DRAM for better performance. A user space migration daemon is being built based on this kernel patchset to make the full vertical solution.

Base kernel is v4.20 . The patches are not suitable for upstreaming in near future -- some are quick hacks, some others need more works. However they are complete enough to demo the necessary kernel changes for the proposed app&VM transparent NVDIMM volatile use model.

The interfaces are far from finalized. They kind of illustrate what would be necessary for creating a user space driven solution. The exact forms will ask for more thoughts and inputs. We may adopt HMAT based solution for NUMA node related interface when they are ready. The /proc/PID/idle_pages interface is standalone but non-trivial. Before upstreaming some day, it's expected to take long time to collect various real use cases and feedbacks, so as to refine and stabilize the format.

Create PMEM numa node

	[PATCH 01/21] e820: cheat PMEM as DRAM

Mark numa node as DRAM/PMEM

[PATCH 02/21] acpi/numa: memorize NUMA node type from SRAT table [PATCH 03/21] x86/numa_emulation: fix fake NUMA in uniform case [PATCH 04/21] x86/numa_emulation: pass numa node type to fake nodes [PATCH 05/21] mmzone: new pgdat flags for DRAM and PMEM [PATCH 06/21] x86,numa: update numa node type
[PATCH 07/21] mm: export node type {pmem|dram} under /sys/bus/node

Point neighbor DRAM/PMEM to each other

[PATCH 08/21] mm: introduce and export pgdat peer_node [PATCH 09/21] mm: avoid duplicate peer target node

Standalone zonelist for DRAM and PMEM nodes

[PATCH 10/21] mm: build separate zonelist for PMEM and DRAM node

Keep page table pages in DRAM

[PATCH 11/21] kvm: allocate page table pages from DRAM [PATCH 12/21] x86/pgtable: allocate page table pages from DRAM

/proc/PID/idle_pages interface for virtual machine and normal tasks

[PATCH 13/21] x86/pgtable: dont check PMD accessed bit [PATCH 14/21] kvm: register in mm_struct [PATCH 15/21] ept-idle: EPT walk for virtual machine [PATCH 16/21] mm-idle: mm_walk for normal task [PATCH 17/21] proc: introduce /proc/PID/idle_pages [PATCH 18/21] kvm-ept-idle: enable module

Mark hot pages

[PATCH 19/21] mm/migrate.c: add move_pages(MPOL_MF_SW_YOUNG) flag

Kernel DRAM=>PMEM migration

[PATCH 20/21] mm/vmscan.c: migrate anon DRAM pages to PMEM node [PATCH 21/21] mm/vmscan.c: shrink anon list if can migrate to PMEM

 arch/x86/include/asm/numa.h    |    2 
 arch/x86/include/asm/pgalloc.h |   10 
 arch/x86/include/asm/pgtable.h |    3 
 arch/x86/kernel/e820.c         |    3 
 arch/x86/kvm/Kconfig           |   11 
 arch/x86/kvm/Makefile          |    4 
 arch/x86/kvm/ept_idle.c        |  841 +++++++++++++++++++++++++++++++
 arch/x86/kvm/ept_idle.h        |  116 ++++
 arch/x86/kvm/mmu.c             |   12 
 arch/x86/mm/numa.c             |    3 
 arch/x86/mm/numa_emulation.c   |   30 +
 arch/x86/mm/pgtable.c          |   22 
 drivers/acpi/numa.c            |    5 
 drivers/base/node.c            |   21 
 fs/proc/base.c                 |    2 
 fs/proc/internal.h             |    1 
 fs/proc/task_mmu.c             |   54 +
 include/linux/mm_types.h       |   11 
 include/linux/mmzone.h         |   38 +
 mm/mempolicy.c                 |   14 
 mm/migrate.c                   |   13 
 mm/page_alloc.c                |   77 ++
 mm/pagewalk.c                  |    1 
 mm/vmscan.c                    |   38 +
 virt/kvm/kvm_main.c            |    3 
 25 files changed, 1306 insertions(+), 29 deletions(-)

V1 patches: https://lkml.org/lkml/2018/9/2/13

Regards,
Fengguang