mm/demotion: Memory tiers and demotion
Subsystems (10)
v13 cover mm/demotion: Memory tiers and demotion
The current kernel has the basic memory tiering support: Inactive pages on a higher tier NUMA node can be migrated (demoted) to a lower tier NUMA node to make room for new allocations on the higher tier NUMA node. Frequently accessed pages on a lower tier NUMA node can be migrated (promoted) to a higher tier NUMA node to improve the performance.
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the highest tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel implementation needs to be improved for several important use cases:
* The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM-backed memory-only node on a virtual machine) and that should be put into a higher tier.
* The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM (e.g. GPU memory) devices, these memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
* Also because the current tier hierarchy always puts CPU nodes into the top tier, when a CPU is hot-added (or hot-removed) and triggers a memory node from CPU-less into a CPU node (or vice versa), the memory tier hierarchy gets changed, even though no memory node is added or removed. This can make the tier hierarchy unstable and make it difficult to support tier-based memory accounting.
* A higher tier node can only be demoted to nodes with shortest distance on the next lower tier as defined by the demotion path, not any other node from any lower tier. This strict, demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), and has resulted in the feature request for an interface to override the system-wide, per-node demotion order from the userspace. This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from any lower tier, whereas the demotion order doesn't allow that.
This patch series make the creation of memory tiers explicit under the control of device driver.
Memory Tier Initialization
==========================
Linux kernel presents memory devices as NUMA nodes and each memory device is of a specific type. The memory type of a device is represented by its abstract distance. A memory tier corresponds to a range of abstract distance. This allows for classifying memory devices with a specific performance range into a memory tier.
By default, all memory nodes are assigned to the default tier with abstract distance 512.
A device driver can move its memory nodes from the default tier. For example, PMEM can move its memory nodes below the default tier, whereas GPU can move its memory nodes above the default tier.
The kernel initialization code makes the decision on which exact tier a memory node should be assigned to based on the requests from the device drivers as well as the memory device hardware information provided by the firmware.
Hot-adding/removing CPUs doesn't affect memory tier hierarchy.
Changes from v12
* Fix kernel crash on module unload
* Address review feedback.
* Add node_random patch to this series based on review feedback
Changes from v11:
* smaller abstract distance imply faster(higher) memory tier.
Changes from v10:
* rename performance level to abstract distance
* Thanks to all the good feedback from Huang, Ying <[email protected]>. Updated the patchset to cover most of the review feedback.
Changes from v9:
* Use performance level for initializing memory tiers.
Changes from v8:
* Drop the sysfs interface patches and related documentation changes.
Changes from v7:
* Fix kernel crash with demotion.
* Improve documentation.
Changes from v6:
* Drop the usage of rank.
* Address other review feedback.
Changes from v5:
* Remove patch supporting N_MEMORY node removal from memory tiers. memory tiers are going to be used for features other than demotion. Hence keep all N_MEMORY nodes in memory tiers irrespective of whether they want to participate in promotion or demotion.
* Add NODE_DATA->memtier
* Rearrage patches to add sysfs files later.
* Add support to create memory tiers from userspace.
* Address other review feedback.
Changes from v4:
* Address review feedback.
* Reverse the meaning of "rank": higher rank value means higher tier.
* Add "/sys/devices/system/memtier/default_tier".
* Add node_is_toptier
v4:
Add support for explicit memory tiers and ranks.
v3:
- Modify patch 1 subject to make it more specific
- Remove /sys/kernel/mm/numa/demotion_targets interface, use /sys/devices/system/node/demotion_targets instead and make it writable to override node_states[N_DEMOTION_TARGETS].
- Add support to view per node demotion targets via sysfs
v2:
In v1, only 1st patch of this patch series was sent, which was implemented to avoid some of the limitations on the demotion target sharing, however for certain numa topology, the demotion targets found by that patch was not most optimal, so 1st patch in this series is modified according to suggestions from Huang and Baolin. Different examples of demotion list comparasion between existing implementation and changed implementation can be found in the commit message of 1st patch.
Aneesh Kumar K.V (8):
mm/demotion: Add support for explicit memory tiers mm/demotion: Move memory demotion related code mm/demotion: Add hotplug callbacks to handle new numa node onlined mm/demotion/dax/kmem: Set node's abstract distance to
MEMTIER_DEFAULT_DAX_ADISTANCE
mm/demotion: Build demotion targets based on explicit memory tiers mm/demotion: Add pg_data_t member to track node memory tier details mm/demotion: Update node_is_toptier to work with memory tiers lib/nodemask: Optimize node_random for nodemask with single NUMA node
Jagdish Gediya (1):
mm/demotion: Demote pages according to allocation fallback order
drivers/dax/kmem.c | 40 ++- include/linux/memory-tiers.h | 84 +++++ include/linux/migrate.h | 15 - include/linux/mmzone.h | 3 + include/linux/node.h | 5 - lib/nodemask.c | 15 +- mm/Makefile | 1 + mm/huge_memory.c | 1 + mm/memory-tiers.c | 616 +++++++++++++++++++++++++++++++++++ mm/migrate.c | 453 +------------------------- mm/mprotect.c | 1 + mm/vmscan.c | 59 +++- mm/vmstat.c | 4 - 13 files changed, 802 insertions(+), 495 deletions(-) create mode 100644 include/linux/memory-tiers.h create mode 100644 mm/memory-tiers.c
Signature · 1 line
2.37.1
PATCH v13 1/9 mm/demotion: Add support for explicit memory tiers
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the highest tier, and builds the tier hierarchy by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel implementation needs to be improved for several important use cases,
The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM-backed memory-only node on a virtual machine) that should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
With current kernel higher tier node can only be demoted to nodes with shortest distance on the next lower tier as defined by the demotion path, not any other node from any lower tier. This strict, demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from any lower tier, whereas the demotion order doesn't allow that.
This patch series address the above by defining memory tiers explicitly.
Linux kernel presents memory devices as NUMA nodes and each memory device is of a specific type. The memory type of a device is represented by its abstract distance. A memory tier corresponds to a range of abstract distance. This allows for classifying memory devices with a specific performance range into a memory tier.
This patch configures the range/chunk size to be 128. The default DRAM abstract distance is 512. We can have 4 memory tiers below the default DRAM with abstract distance range 0 - 127, 127 - 255, 256- 383, 384 - 511. Faster memory devices can be placed in these faster(higher) memory tiers. Slower memory devices like persistent memory will have abstract distance higher than the default DRAM level.
--- include/linux/memory-tiers.h | 15 +++++ mm/Makefile | 1 + mm/memory-tiers.c | 107 +++++++++++++++++++++++++++++++++++ 3 files changed, 123 insertions(+) create mode 100644 include/linux/memory-tiers.h create mode 100644 mm/memory-tiers.c
include/linux/memory-tiers.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_TIERS_H +#define _LINUX_MEMORY_TIERS_H + +/* +* Each tier cover a abstrace distance chunk size of 128 +*/ +#define MEMTIER_CHUNK_BITS 7 +#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS) +/* +* Smaller abstract distance value imply faster(higher) memory tiers. +*/ +#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE) + +#endif /* _LINUX_MEMORY_TIERS_H */
mm/Makefile +1 -0
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
mm/memory-tiers.c new file +107 -0
107 lines not on this page · raw message
Signature · 1 line
2.37.1
Re: [PATCH v13 1/9] mm/demotion: Add support for explicit memory tiers
Quoted 184 lines · Aneesh Kumar K.V
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the highest tier, and builds the tier hierarchy by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel implementation needs to be improved for several important use cases,
The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM-backed memory-only node on a virtual machine) that should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
With current kernel higher tier node can only be demoted to nodes with shortest distance on the next lower tier as defined by the demotion path, not any other node from any lower tier. This strict, demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from any lower tier, whereas the demotion order doesn't allow that.
This patch series address the above by defining memory tiers explicitly.
Linux kernel presents memory devices as NUMA nodes and each memory device is of a specific type. The memory type of a device is represented by its abstract distance. A memory tier corresponds to a range of abstract distance. This allows for classifying memory devices with a specific performance range into a memory tier.
This patch configures the range/chunk size to be 128. The default DRAM abstract distance is 512. We can have 4 memory tiers below the default DRAM with abstract distance range 0 - 127, 127 - 255, 256- 383, 384 - 511. Faster memory devices can be placed in these faster(higher) memory tiers. Slower memory devices like persistent memory will have abstract distance higher than the default DRAM level.
Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/memory-tiers.h | 15 +++++ mm/Makefile | 1 + mm/memory-tiers.c | 107 +++++++++++++++++++++++++++++++++++ 3 files changed, 123 insertions(+) create mode 100644 include/linux/memory-tiers.h create mode 100644 mm/memory-tiers.cinclude/linux/memory-tiers.h new file +15
-0@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_TIERS_H +#define _LINUX_MEMORY_TIERS_H + +/* +* Each tier cover a abstrace distance chunk size of 128 +*/ +#define MEMTIER_CHUNK_BITS 7 +#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS) +/* +* Smaller abstract distance value imply faster(higher) memory tiers. +*/ +#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE) + +#endif /* _LINUX_MEMORY_TIERS_H */ mm/Makefile +1
-0@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.omm/memory-tiers.c new file +91
-0@@ -0,0 +1,107 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/types.h> +#include <linux/nodemask.h> +#include <linux/slab.h> +#include <linux/lockdep.h> +#include <linux/memory-tiers.h> + +struct memory_tier { +/* hierarchy of memory tiers */ +struct list_head list; +/* list of all memory types part of this tier */ +struct list_head memory_types; +/* +* start value of abstract distance. memory tier maps +* an abstract distance range, +* adistance_start .. adistance_start + MEMTIER_CHUNK_SIZE +*/ +int adistance_start; +}; + +struct memory_dev_type { +/* list of memory types that are part of same tier as this type */ +struct list_head tier_sibiling; +/* abstract distance for this specific memory type */ +int adistance; +/* Nodes of same abstract distance */ +nodemask_t nodes; +struct memory_tier *memtier; +}; + +static DEFINE_MUTEX (memory_tier_lock); +static LIST_HEAD (memory_tiers); +static struct memory_dev_type *node_memory_types[MAX_NUMNODES ]; +/* +* For now let's have 4 memory tier below default DRAM tier. +*/ +static struct memory_dev_type default_dram_type = { + .adistance =MEMTIER_ADISTANCE_DRAM , + .tier_sibiling =LIST_HEAD_INIT (default_dram_type.tier_sibiling), +}; + +static struct memory_tier *find_create_memory_tier (struct memory_dev_type *memtype) +{ +bool found_slot = false; +struct memory_tier *memtier, *new_memtier; +int adistance = memtype->adistance; +unsigned int memtier_adistance_chunk_size =MEMTIER_CHUNK_SIZE ; + +lockdep_assert_held_once (&memory_tier_lock); + +/* +* If the memtype is already part of a memory tier, +* just return that. +*/ +if (memtype->memtier) +return memtype->memtier; + + adistance =round_down (adistance, memtier_adistance_chunk_size); +list_for_each_entry (memtier, &memory_tiers, list) { +if (adistance == memtier->adistance_start) { + memtype->memtier = memtier; +list_add (&memtype->tier_sibiling, &memtier->memory_types); +return memtier; + }else if (adistance < memtier->adistance_start) { + found_slot = true; +break ; + } + } + + new_memtier =kmalloc (sizeof (struct memory_tier ),GFP_KERNEL ); +if (!new_memtier) +return ERR_PTR (-ENOMEM ); + + new_memtier->adistance_start = adistance; +INIT_LIST_HEAD (&new_memtier->list); +INIT_LIST_HEAD (&new_memtier->memory_types); +if (found_slot) +list_add_tail (&new_memtier->list, &memtier->list); +else +list_add_tail (&new_memtier->list, &memory_tiers); + memtype->memtier = new_memtier; +list_add (&memtype->tier_sibiling, &new_memtier->memory_types); +return new_memtier; +} + +static int __init memory_tier_init (void ) +{ +int node; +struct memory_tier *memtier; + +mutex_lock (&memory_tier_lock);
mm/memory-tiers.c line 101
Quoted 10 lines
mm/memory-tiers.c new file +10
-0+/* CPU only nodes are not part of memory tiers. */ + default_dram_type.nodes = node_states[N_MEMORY ]; + + memtier =find_create_memory_tier (&default_dram_type); +if (IS_ERR (memtier)) +panic ("%s() failed to register memory tier: %ld\n" , + __func__,PTR_ERR (memtier)); + +for_each_node_state (node,N_MEMORY ) + node_memory_types[node] = &default_dram_type;
Although not absolutely necessary, it seems better to set node_memory_types[] before adding nodes to the memory type and adding the memory type to the memory tier.
Best Regards,
Huang, Ying
Quoted 6 lines
mm/memory-tiers.c new file +6
-0+ +mutex_unlock (&memory_tier_lock); + +return 0 ; +} +subsys_initcall (memory_tier_init);
Re: [PATCH v13 1/9] mm/demotion: Add support for explicit memory tiers
Quoted 192 lines · Ying
Quoted 190 lines · Aneesh Kumar K.V
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the highest tier, and builds the tier hierarchy by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel implementation needs to be improved for several important use cases,
The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM-backed memory-only node on a virtual machine) that should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
With current kernel higher tier node can only be demoted to nodes with shortest distance on the next lower tier as defined by the demotion path, not any other node from any lower tier. This strict, demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from any lower tier, whereas the demotion order doesn't allow that.
This patch series address the above by defining memory tiers explicitly.
Linux kernel presents memory devices as NUMA nodes and each memory device is of a specific type. The memory type of a device is represented by its abstract distance. A memory tier corresponds to a range of abstract distance. This allows for classifying memory devices with a specific performance range into a memory tier.
This patch configures the range/chunk size to be 128. The default DRAM abstract distance is 512. We can have 4 memory tiers below the default DRAM with abstract distance range 0 - 127, 127 - 255, 256- 383, 384 - 511. Faster memory devices can be placed in these faster(higher) memory tiers. Slower memory devices like persistent memory will have abstract distance higher than the default DRAM level.
Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/memory-tiers.h | 15 +++++ mm/Makefile | 1 + mm/memory-tiers.c | 107 +++++++++++++++++++++++++++++++++++ 3 files changed, 123 insertions(+) create mode 100644 include/linux/memory-tiers.h create mode 100644 mm/memory-tiers.cinclude/linux/memory-tiers.h new file +15
-0@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_TIERS_H +#define _LINUX_MEMORY_TIERS_H + +/* +* Each tier cover a abstrace distance chunk size of 128 +*/ +#define MEMTIER_CHUNK_BITS 7 +#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS) +/* +* Smaller abstract distance value imply faster(higher) memory tiers. +*/ +#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE) + +#endif /* _LINUX_MEMORY_TIERS_H */ mm/Makefile +1
-0@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.omm/memory-tiers.c new file +97
-0@@ -0,0 +1,107 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/types.h> +#include <linux/nodemask.h> +#include <linux/slab.h> +#include <linux/lockdep.h> +#include <linux/memory-tiers.h> + +struct memory_tier { +/* hierarchy of memory tiers */ +struct list_head list; +/* list of all memory types part of this tier */ +struct list_head memory_types; +/* +* start value of abstract distance. memory tier maps +* an abstract distance range, +* adistance_start .. adistance_start + MEMTIER_CHUNK_SIZE +*/ +int adistance_start; +}; + +struct memory_dev_type { +/* list of memory types that are part of same tier as this type */ +struct list_head tier_sibiling; +/* abstract distance for this specific memory type */ +int adistance; +/* Nodes of same abstract distance */ +nodemask_t nodes; +struct memory_tier *memtier; +}; + +static DEFINE_MUTEX (memory_tier_lock); +static LIST_HEAD (memory_tiers); +static struct memory_dev_type *node_memory_types[MAX_NUMNODES ]; +/* +* For now let's have 4 memory tier below default DRAM tier. +*/ +static struct memory_dev_type default_dram_type = { + .adistance =MEMTIER_ADISTANCE_DRAM , + .tier_sibiling =LIST_HEAD_INIT (default_dram_type.tier_sibiling), +}; + +static struct memory_tier *find_create_memory_tier (struct memory_dev_type *memtype) +{ +bool found_slot = false; +struct memory_tier *memtier, *new_memtier; +int adistance = memtype->adistance; +unsigned int memtier_adistance_chunk_size =MEMTIER_CHUNK_SIZE ; + +lockdep_assert_held_once (&memory_tier_lock); + +/* +* If the memtype is already part of a memory tier, +* just return that. +*/ +if (memtype->memtier) +return memtype->memtier; + + adistance =round_down (adistance, memtier_adistance_chunk_size); +list_for_each_entry (memtier, &memory_tiers, list) { +if (adistance == memtier->adistance_start) { + memtype->memtier = memtier; +list_add (&memtype->tier_sibiling, &memtier->memory_types); +return memtier; + }else if (adistance < memtier->adistance_start) { + found_slot = true; +break ; + } + } + + new_memtier =kmalloc (sizeof (struct memory_tier ),GFP_KERNEL ); +if (!new_memtier) +return ERR_PTR (-ENOMEM ); + + new_memtier->adistance_start = adistance; +INIT_LIST_HEAD (&new_memtier->list); +INIT_LIST_HEAD (&new_memtier->memory_types); +if (found_slot) +list_add_tail (&new_memtier->list, &memtier->list); +else +list_add_tail (&new_memtier->list, &memory_tiers); + memtype->memtier = new_memtier; +list_add (&memtype->tier_sibiling, &new_memtier->memory_types); +return new_memtier; +} + +static int __init memory_tier_init (void ) +{ +int node; +struct memory_tier *memtier; + +mutex_lock (&memory_tier_lock); +/* CPU only nodes are not part of memory tiers. */ + default_dram_type.nodes = node_states[N_MEMORY ]; + + memtier =find_create_memory_tier (&default_dram_type); +if (IS_ERR (memtier)) + panic("%s() failed to register memory tier: %ld\n",
Quoted 10 lines
mm/memory-tiers.c line 101
Quoted 4 lines
mm/memory-tiers.c new file +4
-0+ __func__, PTR_ERR(memtier)); + +for_each_node_state (node,N_MEMORY ) + node_memory_types[node] = &default_dram_type;Although not absolutely necessary, it seems better to set node_memory_types[] before adding nodes to the memory type and adding the memory type to the memory tier.
updated.
-aneesh
PATCH v13 2/9 mm/demotion: Move memory demotion related code
In reply to Aneesh Kumar K.VThis move memory demotion related code to mm/memory-tiers.c. No functional change in this patch.
--- include/linux/memory-tiers.h | 8 +++++ include/linux/migrate.h | 2 -- mm/memory-tiers.c | 64 ++++++++++++++++++++++++++++++++++++ mm/migrate.c | 60 +-------------------------------- mm/vmscan.c | 1 + 5 files changed, 74 insertions(+), 61 deletions(-)
include/linux/memory-tiers.h +8 -0
@@ -12,4 +12,12 @@*/ #define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE) +#ifdef CONFIG_NUMA +#include <linux/types.h> +extern bool numa_demotion_enabled; + +#else + +#define numa_demotion_enabled false +#endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */
include/linux/migrate.h +0 -2
@@ -103,7 +103,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#if defined(CONFIG_MIGRATION ) && defined(CONFIG_NUMA )extern void set_migration_target_nodes (void );extern void migrate_on_reclaim_init (void ); -extern bool numa_demotion_enabled;extern int next_demotion_node (int node);#else static inline void set_migration_target_nodes (void ) {}
@@ -112,7 +111,6 @@ static inline int next_demotion_node(int node) {return NUMA_NO_NODE ; } -#define numa_demotion_enabled false#endif #ifdef CONFIG_COMPACTION
mm/memory-tiers.c +64 -0
73 lines not on this page · raw message
mm/migrate.c +1 -59
65 lines not on this page · raw message
mm/vmscan.c +1 -0
@@ -50,6 +50,7 @@#include <linux/printk.h> #include <linux/dax.h> #include <linux/psi.h> +#include <linux/memory-tiers.h> #include <asm/tlbflush.h> #include <asm/div64.h>
Signature · 1 line
2.37.1
PATCH v13 3/9 mm/demotion: Add hotplug callbacks to handle new numa node onlined
In reply to Aneesh Kumar K.VIf the new NUMA node onlined doesn't have a abstract distance assigned, the kernel adds the NUMA node to default memory tier.
--- include/linux/memory-tiers.h | 1 + mm/memory-tiers.c | 87 ++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+)
include/linux/memory-tiers.h +1 -0
@@ -11,6 +11,7 @@* Smaller abstract distance value imply faster(higher) memory tiers. */ #define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE) +#define MEMTIER_HOTPLUG_PRIO 100#ifdef CONFIG_NUMA #include <linux/types.h>
mm/memory-tiers.c +87 -0
105 lines not on this page · raw message
Signature · 1 line
2.37.1
PATCH v13 4/9 mm/demotion/dax/kmem: Set node's abstract distance to MEMTIER_DEFAULT_DAX_ADISTANCE
In reply to Aneesh Kumar K.VBy default, all nodes are assigned to the default memory tier which is the memory tier designated for nodes with DRAM
Set dax kmem device node's tier to slower memory tier by assigning abstract distance to MEMTIER_DEFAULT_DAX_ADISTANCE. Low-level drivers like papr_scm or ACPI NFIT can initialize memory device type to a more accurate value based on device tree details or HMAT. If the kernel doesn't find the memory type initialized, a default slower memory type is assigned by the kmem driver.
--- drivers/dax/kmem.c | 40 ++++++++++++++++++- include/linux/memory-tiers.h | 26 ++++++++++++- mm/memory-tiers.c | 74 +++++++++++++++++++++++++----------- 3 files changed, 115 insertions(+), 25 deletions(-)
drivers/dax/kmem.c +39 -1
@@ -11,9 +11,17 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mman.h> +#include <linux/memory-tiers.h> #include "dax-private.h" #include "bus.h" +/* +* Default abstract distance assigned to the NUMA node onlined +* by DAX/kmem if the low level platform driver didn't initialize +* one for this NUMA node. +*/ +#define MEMTIER_DEFAULT_DAX_ADISTANCE (MEMTIER_ADISTANCE_DRAM * 2) +/* Memory resource name used for add_memory_driver_managed(). */ static const char *kmem_name;/* Set if any memory will remain added when the driver will be unloaded. */
@@ -41,6 +49,7 @@ struct dax_kmem_data {struct resource *res[]; }; +static struct memory_dev_type *dax_slowmem_type;static int dev_dax_kmem_probe (struct dev_dax *dev_dax) {struct device *dev = &dev_dax->dev;
@@ -62,6 +71,8 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)return -EINVAL ; } +init_node_memory_type (numa_node, dax_slowmem_type); +for (i =0 ; i < dev_dax->nr_range; i++) {struct range range;
@@ -162,6 +173,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)static void dev_dax_kmem_remove (struct dev_dax *dev_dax) {int i, success =0 ; +int node = dev_dax->target_node;struct device *dev = &dev_dax->dev;struct dax_kmem_data *data =dev_get_drvdata (dev);
@@ -198,6 +210,14 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)kfree (data->res_name);kfree (data);dev_set_drvdata (dev,NULL ); +/* +* Clear the memtype association on successful unplug. +* If not, we have memory blocks left which can be +* offlined/onlined later. We need to keep memory_dev_type +* for that. This implies this reference will be around +* till next reboot. +*/ + clear_node_memory_type(node, dax_slowmem_type); } }#else
@@ -228,9 +248,27 @@ static int __init dax_kmem_init(void)if (!kmem_name)return -ENOMEM ; + dax_slowmem_type =kmalloc (sizeof (*dax_slowmem_type),GFP_KERNEL ); +if (!dax_slowmem_type) { + rc = -ENOMEM ; + goto kmem_name_free; + } + dax_slowmem_type->adistance =MEMTIER_DEFAULT_DAX_ADISTANCE ; +INIT_LIST_HEAD (&dax_slowmem_type->tier_sibiling); + dax_slowmem_type->nodes =NODE_MASK_NONE ; + dax_slowmem_type->memtier =NULL ; +kref_init (&dax_slowmem_type->kref); + rc =dax_driver_register (&device_dax_kmem_driver);if (rc) -kfree_const (kmem_name); + goto error_out; + +return rc; + +error_out: +kfree (dax_slowmem_type); +kmem_name_free: +kfree_const (kmem_name);return rc; }
include/linux/memory-tiers.h +25 -1
@@ -2,6 +2,8 @@#ifndef _LINUX_MEMORY_TIERS_H#define _LINUX_MEMORY_TIERS_H +#include <linux/types.h> +#include <linux/nodemask.h> /* * Each tier cover a abstrace distance chunk size of 128 */
@@ -13,12 +15,34 @@#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE)#define MEMTIER_HOTPLUG_PRIO 100 +struct memory_tier ; +struct memory_dev_type { +/* list of memory types that are part of same tier as this type */ +struct list_head tier_sibiling; +/* abstract distance for this specific memory type */ +int adistance; +/* Nodes of same abstract distance */ +nodemask_t nodes; +struct kref kref; +struct memory_tier *memtier; +}; +#ifdef CONFIG_NUMA -#include <linux/types.h> extern bool numa_demotion_enabled; +void init_node_memory_type (int node,struct memory_dev_type *default_type); +void clear_node_memory_type (int node,struct memory_dev_type *memtype);#else #define numa_demotion_enabled false +static inline void init_node_memory_type (int node,struct memory_dev_type *default_type) +{ + +} + +static inline void clear_node_memory_type (int node,struct memory_dev_type *memtype) +{ + +}#endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */
mm/memory-tiers.c +51 -23
119 lines not on this page · raw message
Signature · 1 line
2.37.1
Re: [PATCH v13 4/9] mm/demotion/dax/kmem: Set node's abstract distance to MEMTIER_DEFAULT_DAX_ADISTANCE
Quoted 7 lines · Aneesh Kumar K.V
By default, all nodes are assigned to the default memory tier which is the memory tier designated for nodes with DRAM
Set dax kmem device node's tier to slower memory tier by assigning abstract distance to
MEMTIER_DEFAULT_DAX_ADISTANCE. Low-level drivers likepapr_scmor ACPI NFIT can initialize memory device type to a more accurate value based on device tree details or HMAT.
I don't know how ACPI NFIT can help here. Can you teach me?
Per my understanding, we may use the information provided by ACPI SLIT or HMAT (or device tree via papr_scm) to create memory types. Before that is implemented, we just create a memory type with default abstract distance.
Quoted 18 lines
If the
kernel doesn't find the memory type initialized, a default slower memory type is assigned by the kmem driver.Signed-off-by Aneesh Kumar K.V <[email protected]>--- drivers/dax/kmem.c | 40 ++++++++++++++++++- include/linux/memory-tiers.h | 26 ++++++++++++- mm/memory-tiers.c | 74 +++++++++++++++++++++++++----------- 3 files changed, 115 insertions(+), 25 deletions(-)drivers/dax/kmem.c +0
-0@@ -11,9 +11,17 @@#include <linux/fs.h> #include <linux/mm.h>
drivers/dax/kmem.c line 22
Quoted 10 lines
drivers/dax/kmem.c +6
-0#include <linux/mman.h> +#include <linux/memory-tiers.h> #include "dax-private.h" #include "bus.h" +/* +* Default abstract distance assigned to the NUMA node onlined +* by DAX/kmem if the low level platform driver didn't initialize +* one for this NUMA node. +*/
We have 2 choices here.
1. The low level drivers create memory types and set node_memory_types[]. We need a mechanism to coordinate among multiple drivers. On x86, we have ACPI SLIT, HMAT. And we have platform independent CXL defined CDAT.
2. The high level driver (such as dax/kmem.c) coordinate among multiple low level drivers. For example, it may query CXL CDAT firstly, and use a notifier chain to query platform drivers with priority.
Personally, I prefer choice 2. We can discuss this later. But can we make the comments more general to avoid to make decision now?
drivers/dax/kmem.c struct dax_kmem_data { · line 52
Quoted 10 lines
drivers/dax/kmem.c +3
-0+#define MEMTIER_DEFAULT_DAX_ADISTANCE (MEMTIER_ADISTANCE_DRAM * 2) +/* Memory resource name used for add_memory_driver_managed(). */ static const char *kmem_name;/* Set if any memory will remain added when the driver will be unloaded. */ @@ -41,6 +49,7 @@ struct dax_kmem_data {struct resource *res[]; }; +static struct memory_dev_type *dax_slowmem_type;
I don't think "slowmem" make much sense here. There may be even slower memory. Just dax_mem_type or dax_kmem_type should be OK?
drivers/dax/kmem.c static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · line 75
Quoted 9 lines
drivers/dax/kmem.c +2
-0static int dev_dax_kmem_probe (struct dev_dax *dev_dax) {struct device *dev = &dev_dax->dev;@@ -62,6 +71,8 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)return -EINVAL ; } +init_node_memory_type (numa_node, dax_slowmem_type); +
I don't find error handling in this function. Per my understanding, if memory hot-add fails, we need to call clear_node_memory_type().
Quoted 31 lines
drivers/dax/kmem.c +10
-0for (i =0 ; i < dev_dax->nr_range; i++) {struct range range;@@ -162,6 +173,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)static void dev_dax_kmem_remove (struct dev_dax *dev_dax) {int i, success =0 ; +int node = dev_dax->target_node;struct device *dev = &dev_dax->dev;struct dax_kmem_data *data =dev_get_drvdata (dev);@@ -198,6 +210,14 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)kfree (data->res_name);kfree (data);dev_set_drvdata (dev,NULL ); +/* +* Clear the memtype association on successful unplug. +* If not, we have memory blocks left which can be +* offlined/onlined later. We need to keep memory_dev_type +* for that. This implies this reference will be around +* till next reboot. +*/ + clear_node_memory_type(node, dax_slowmem_type); } }#else @@ -228,9 +248,27 @@ static int __init dax_kmem_init(void)if (!kmem_name)return -ENOMEM ; + dax_slowmem_type =kmalloc (sizeof (*dax_slowmem_type),GFP_KERNEL );
drivers/dax/kmem.c static int __init dax_kmem_init(void) · line 261
Quoted 10 lines
drivers/dax/kmem.c +10
-0+if (!dax_slowmem_type) { + rc = -ENOMEM ; + goto kmem_name_free; + } + dax_slowmem_type->adistance =MEMTIER_DEFAULT_DAX_ADISTANCE ; +INIT_LIST_HEAD (&dax_slowmem_type->tier_sibiling); + dax_slowmem_type->nodes =NODE_MASK_NONE ; + dax_slowmem_type->memtier =NULL ; +kref_init (&dax_slowmem_type->kref); +
Here we initialize the kref to 1. So in dax_kmem_exit() we should drop the last reference, otherwise we cannot free the memory type?
Quoted 166 lines
drivers/dax/kmem.c +8
-1rc =dax_driver_register (&device_dax_kmem_driver);if (rc) -kfree_const (kmem_name); + goto error_out; + +return rc; + +error_out: +kfree (dax_slowmem_type); +kmem_name_free: +kfree_const (kmem_name);return rc; }include/linux/memory-tiers.h +25
-1@@ -2,6 +2,8 @@#ifndef _LINUX_MEMORY_TIERS_H#define _LINUX_MEMORY_TIERS_H +#include <linux/types.h> +#include <linux/nodemask.h> /* * Each tier cover a abstrace distance chunk size of 128 */ @@ -13,12 +15,34 @@#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE)#define MEMTIER_HOTPLUG_PRIO 100 +struct memory_tier ; +struct memory_dev_type { +/* list of memory types that are part of same tier as this type */ +struct list_head tier_sibiling; +/* abstract distance for this specific memory type */ +int adistance; +/* Nodes of same abstract distance */ +nodemask_t nodes; +struct kref kref; +struct memory_tier *memtier; +}; +#ifdef CONFIG_NUMA -#include <linux/types.h> extern bool numa_demotion_enabled; +void init_node_memory_type (int node,struct memory_dev_type *default_type); +void clear_node_memory_type (int node,struct memory_dev_type *memtype);#else #define numa_demotion_enabled false +static inline void init_node_memory_type (int node,struct memory_dev_type *default_type) +{ + +} + +static inline void clear_node_memory_type (int node,struct memory_dev_type *memtype) +{ + +}#endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */ mm/memory-tiers.c +39
-20@@ -1,6 +1,4 @@// SPDX-License-Identifier: GPL-2.0 -#include <linux/types.h> -#include <linux/nodemask.h> #include <linux/slab.h> #include <linux/lockdep.h> #include <linux/sysfs.h> @@ -21,26 +19,10 @@ struct memory_tier {int adistance_start; }; -struct memory_dev_type { -/* list of memory types that are part of same tier as this type */ -struct list_head tier_sibiling; -/* abstract distance for this specific memory type */ -int adistance; -/* Nodes of same abstract distance */ -nodemask_t nodes; -struct memory_tier *memtier; -}; -static DEFINE_MUTEX (memory_tier_lock);static LIST_HEAD (memory_tiers);static struct memory_dev_type *node_memory_types[MAX_NUMNODES ]; -/* -* For now let's have 4 memory tier below default DRAM tier. -*/ -static struct memory_dev_type default_dram_type = { - .adistance =MEMTIER_ADISTANCE_DRAM , - .tier_sibiling =LIST_HEAD_INIT (default_dram_type.tier_sibiling), -}; +static struct memory_dev_type *default_dram_type;static struct memory_tier *find_create_memory_tier (struct memory_dev_type *memtype) {@@ -96,6 +78,14 @@ static struct memory_tier *__node_get_memory_tier(int node)return NULL ; } +static inline void __init_node_memory_type (int node,struct memory_dev_type *default_type) +{ +if (!node_memory_types[node]) { + node_memory_types[node] = default_type; +kref_get (&default_type->kref); + } +} +static struct memory_tier *set_node_memory_tier (int node) {struct memory_tier *memtier;@@ -107,7 +97,7 @@ static struct memory_tier *set_node_memory_tier(int node)return ERR_PTR (-EINVAL );if (!node_memory_types[node]) - node_memory_types[node] = &default_dram_type; +__init_node_memory_type (node, default_dram_type); memtype = node_memory_types[node];node_set (node, memtype->nodes);@@ -143,6 +133,34 @@ static bool clear_node_memory_tier(int node)return cleared; } +void init_node_memory_type (int node,struct memory_dev_type *default_type) +{ + +mutex_lock (&memory_tier_lock); +__init_node_memory_type (node, default_type); +mutex_unlock (&memory_tier_lock); +} +EXPORT_SYMBOL_GPL (init_node_memory_type); + +static void release_memtype (struct kref *kref) +{ +struct memory_dev_type *memtype; + + memtype = container_of(kref, struct memory_dev_type, kref); +kfree (memtype); +} + +void clear_node_memory_type (int node,struct memory_dev_type *memtype) +{ +mutex_lock (&memory_tier_lock); +if (node_memory_types[node] == memtype) { + node_memory_types[node] =NULL ; +kref_put (&memtype->kref, release_memtype); + } +mutex_unlock (&memory_tier_lock); +} +EXPORT_SYMBOL_GPL (clear_node_memory_type); +static int __meminit memtier_hotplug_callback (struct notifier_block *self,unsigned long action,void *_arg) {@@ -176,17 +194,27 @@ static int __init memory_tier_init(void)int node;struct memory_tier *memtier; + default_dram_type =kmalloc (sizeof (*default_dram_type),GFP_KERNEL );
mm/memory-tiers.c static int __init memory_tier_init(void) · line 207
Quoted 10 lines
mm/memory-tiers.c +9
-0+if (!default_dram_type) +panic ("%s() failed to allocate default DRAM tier\n" , __func__); +mutex_lock (&memory_tier_lock); + +/* For now let's have 4 memory tier below default DRAM tier. */ + default_dram_type->adistance =MEMTIER_ADISTANCE_DRAM ; +INIT_LIST_HEAD (&default_dram_type->tier_sibiling); + default_dram_type->memtier =NULL ; +kref_init (&default_dram_type->kref);
It appears that we can define a function to initialize a memory type.
Quoted 5 lines
mm/memory-tiers.c +1
-2/* CPU only nodes are not part of memory tiers. */ - default_dram_type.nodes = node_states[N_MEMORY ]; + default_dram_type->nodes = node_states[N_MEMORY ]; - memtier =find_create_memory_tier (&default_dram_type);
mm/memory-tiers.c static int __init memory_tier_init(void) · line 219
Quoted 10 lines
mm/memory-tiers.c +2
-1+ memtier =find_create_memory_tier (default_dram_type);if (IS_ERR (memtier))panic ("%s() failed to register memory tier: %ld\n" , __func__,PTR_ERR (memtier));for_each_node_state (node,N_MEMORY ) - node_memory_types[node] = &default_dram_type; +__init_node_memory_type (node, default_dram_type);mutex_unlock (&memory_tier_lock);
Best Regards,
Huang, Ying
Re: [PATCH v13 4/9] mm/demotion/dax/kmem: Set node's abstract distance to MEMTIER_DEFAULT_DAX_ADISTANCE
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K.V
By default, all nodes are assigned to the default memory tier which is the memory tier designated for nodes with DRAM
Set dax kmem device node's tier to slower memory tier by assigning abstract distance to
MEMTIER_DEFAULT_DAX_ADISTANCE. Low-level drivers likepapr_scmor ACPI NFIT can initialize memory device type to a more accurate value based on device tree details or HMAT.
Quoted 7 lines
I don't know how ACPI NFIT can help here. Can you teach me?
Per my understanding, we may use the information provided by ACPI SLIT or HMAT (or device tree via
papr_scm) to create memory types. Before that is implemented, we just create a memory type with default abstract distance.
My idea is to use ACPI NFIT driver that creates a persistent memory region (nvdimm_region_create) to also create memory type and assign that to the NUMA node mapping that region. For now NFIT driver manages all the persistent memory region/DIMM creation (drivers/acpi/nfit/core.c). It can also do the memory type creation using other ACPI information like SLIT/HMAT etc. Similarly CXL driver can do the same using CDAT.
Quoted 52 lines
Quoted 18 lines
If the
kernel doesn't find the memory type initialized, a default slower memory type is assigned by the kmem driver.Signed-off-by Aneesh Kumar K.V <[email protected]>--- drivers/dax/kmem.c | 40 ++++++++++++++++++- include/linux/memory-tiers.h | 26 ++++++++++++- mm/memory-tiers.c | 74 +++++++++++++++++++++++++----------- 3 files changed, 115 insertions(+), 25 deletions(-)drivers/dax/kmem.c +0
-0@@ -11,9 +11,17 @@#include <linux/fs.h> #include <linux/mm.h> drivers/dax/kmem.c line 22
Quoted 10 lines
drivers/dax/kmem.c +6
-0#include <linux/mman.h> +#include <linux/memory-tiers.h> #include "dax-private.h" #include "bus.h" +/* +* Default abstract distance assigned to the NUMA node onlined +* by DAX/kmem if the low level platform driver didn't initialize +* one for this NUMA node. +*/ We have 2 choices here.
1. The low level drivers create memory types and set node_memory_types[]. We need a mechanism to coordinate among multiple drivers. On x86, we have ACPI SLIT, HMAT. And we have platform independent CXL defined CDAT.
2. The high level driver (such as
dax/kmem.c) coordinate among multiple low level drivers. For example, it may query CXL CDAT firstly, and use a notifier chain to query platform drivers with priority.Personally, I prefer choice 2. We can discuss this later. But can we make the comments more general to avoid to make decision now?
Quoted 9 lines
drivers/dax/kmem.c +2
-0+#define MEMTIER_DEFAULT_DAX_ADISTANCE (MEMTIER_ADISTANCE_DRAM * 2) +/* Memory resource name used for add_memory_driver_managed(). */ static const char *kmem_name;/* Set if any memory will remain added when the driver will be unloaded. */ @@ -41,6 +49,7 @@ struct dax_kmem_data {struct resource *res[]; };
Quoted 5 lines
drivers/dax/kmem.c struct dax_kmem_data { · line 52
Quoted 1 line
drivers/dax/kmem.c +1
-0+static struct memory_dev_type *dax_slowmem_type;I don't think "slowmem" make much sense here. There may be even slower memory. Just
dax_mem_typeordax_kmem_typeshould be OK?
Dan Williams had a question about why we are adding an abstract distance slower than DRAM in dax/kmem where we can also initialize faster than DRAM memory using kmem. The "slowmem" was added to indicate that this type is slower than DRAM.
Quoted 7 lines
Quoted 7 lines
drivers/dax/kmem.c +0
-0static int dev_dax_kmem_probe (struct dev_dax *dev_dax) {struct device *dev = &dev_dax->dev;@@ -62,6 +71,8 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)return -EINVAL ; }
Quoted 6 lines
drivers/dax/kmem.c static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · line 75
Quoted 2 lines
drivers/dax/kmem.c +2
-0+init_node_memory_type (numa_node, dax_slowmem_type); +I don't find error handling in this function. Per my understanding, if memory hot-add fails, we need to call
clear_node_memory_type().
Added
Quoted 30 lines
Quoted 30 lines
drivers/dax/kmem.c +9
-0for (i =0 ; i < dev_dax->nr_range; i++) {struct range range;@@ -162,6 +173,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)static void dev_dax_kmem_remove (struct dev_dax *dev_dax) {int i, success =0 ; +int node = dev_dax->target_node;struct device *dev = &dev_dax->dev;struct dax_kmem_data *data =dev_get_drvdata (dev);@@ -198,6 +210,14 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)kfree (data->res_name);kfree (data);dev_set_drvdata (dev,NULL ); +/* +* Clear the memtype association on successful unplug. +* If not, we have memory blocks left which can be +* offlined/onlined later. We need to keep memory_dev_type +* for that. This implies this reference will be around +* till next reboot. +*/ + clear_node_memory_type(node, dax_slowmem_type); } }#else @@ -228,9 +248,27 @@ static int __init dax_kmem_init(void)if (!kmem_name)return -ENOMEM ;
Quoted 15 lines
drivers/dax/kmem.c static int __init dax_kmem_init(void) · line 261
Quoted 11 lines
drivers/dax/kmem.c +11
-0+ dax_slowmem_type =kmalloc (sizeof (*dax_slowmem_type),GFP_KERNEL ); +if (!dax_slowmem_type) { + rc = -ENOMEM ; + goto kmem_name_free; + } + dax_slowmem_type->adistance =MEMTIER_DEFAULT_DAX_ADISTANCE ; +INIT_LIST_HEAD (&dax_slowmem_type->tier_sibiling); + dax_slowmem_type->nodes =NODE_MASK_NONE ; + dax_slowmem_type->memtier =NULL ; +kref_init (&dax_slowmem_type->kref); +Here we initialize the kref to 1. So in
dax_kmem_exit()we should drop the last reference, otherwise we cannot free the memory type?
Add the required destroy to dax_kmem_exit()
Quoted 165 lines
Quoted 165 lines
drivers/dax/kmem.c +8
-1rc =dax_driver_register (&device_dax_kmem_driver);if (rc) -kfree_const (kmem_name); + goto error_out; + +return rc; + +error_out: +kfree (dax_slowmem_type); +kmem_name_free: +kfree_const (kmem_name);return rc; }include/linux/memory-tiers.h +25
-1@@ -2,6 +2,8 @@#ifndef _LINUX_MEMORY_TIERS_H#define _LINUX_MEMORY_TIERS_H +#include <linux/types.h> +#include <linux/nodemask.h> /* * Each tier cover a abstrace distance chunk size of 128 */ @@ -13,12 +15,34 @@#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE)#define MEMTIER_HOTPLUG_PRIO 100 +struct memory_tier ; +struct memory_dev_type { +/* list of memory types that are part of same tier as this type */ +struct list_head tier_sibiling; +/* abstract distance for this specific memory type */ +int adistance; +/* Nodes of same abstract distance */ +nodemask_t nodes; +struct kref kref; +struct memory_tier *memtier; +}; +#ifdef CONFIG_NUMA -#include <linux/types.h> extern bool numa_demotion_enabled; +void init_node_memory_type (int node,struct memory_dev_type *default_type); +void clear_node_memory_type (int node,struct memory_dev_type *memtype);#else #define numa_demotion_enabled false +static inline void init_node_memory_type (int node,struct memory_dev_type *default_type) +{ + +} + +static inline void clear_node_memory_type (int node,struct memory_dev_type *memtype) +{ + +}#endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */ mm/memory-tiers.c +38
-20@@ -1,6 +1,4 @@// SPDX-License-Identifier: GPL-2.0 -#include <linux/types.h> -#include <linux/nodemask.h> #include <linux/slab.h> #include <linux/lockdep.h> #include <linux/sysfs.h> @@ -21,26 +19,10 @@ struct memory_tier {int adistance_start; }; -struct memory_dev_type { -/* list of memory types that are part of same tier as this type */ -struct list_head tier_sibiling; -/* abstract distance for this specific memory type */ -int adistance; -/* Nodes of same abstract distance */ -nodemask_t nodes; -struct memory_tier *memtier; -}; -static DEFINE_MUTEX (memory_tier_lock);static LIST_HEAD (memory_tiers);static struct memory_dev_type *node_memory_types[MAX_NUMNODES ]; -/* -* For now let's have 4 memory tier below default DRAM tier. -*/ -static struct memory_dev_type default_dram_type = { - .adistance =MEMTIER_ADISTANCE_DRAM , - .tier_sibiling =LIST_HEAD_INIT (default_dram_type.tier_sibiling), -}; +static struct memory_dev_type *default_dram_type;static struct memory_tier *find_create_memory_tier (struct memory_dev_type *memtype) {@@ -96,6 +78,14 @@ static struct memory_tier *__node_get_memory_tier(int node)return NULL ; } +static inline void __init_node_memory_type (int node,struct memory_dev_type *default_type) +{ +if (!node_memory_types[node]) { + node_memory_types[node] = default_type; +kref_get (&default_type->kref); + } +} +static struct memory_tier *set_node_memory_tier (int node) {struct memory_tier *memtier;@@ -107,7 +97,7 @@ static struct memory_tier *set_node_memory_tier(int node)return ERR_PTR (-EINVAL );if (!node_memory_types[node]) - node_memory_types[node] = &default_dram_type; +__init_node_memory_type (node, default_dram_type); memtype = node_memory_types[node];node_set (node, memtype->nodes);@@ -143,6 +133,34 @@ static bool clear_node_memory_tier(int node)return cleared; } +void init_node_memory_type (int node,struct memory_dev_type *default_type) +{ + +mutex_lock (&memory_tier_lock); +__init_node_memory_type (node, default_type); +mutex_unlock (&memory_tier_lock); +} +EXPORT_SYMBOL_GPL (init_node_memory_type); + +static void release_memtype (struct kref *kref) +{ +struct memory_dev_type *memtype; + + memtype = container_of(kref, struct memory_dev_type, kref); +kfree (memtype); +} + +void clear_node_memory_type (int node,struct memory_dev_type *memtype) +{ +mutex_lock (&memory_tier_lock); +if (node_memory_types[node] == memtype) { + node_memory_types[node] =NULL ; +kref_put (&memtype->kref, release_memtype); + } +mutex_unlock (&memory_tier_lock); +} +EXPORT_SYMBOL_GPL (clear_node_memory_type); +static int __meminit memtier_hotplug_callback (struct notifier_block *self,unsigned long action,void *_arg) {@@ -176,17 +194,27 @@ static int __init memory_tier_init(void)int node;struct memory_tier *memtier;
Quoted 13 lines
mm/memory-tiers.c static int __init memory_tier_init(void) · line 207
Quoted 11 lines
mm/memory-tiers.c +10
-0+ default_dram_type =kmalloc (sizeof (*default_dram_type),GFP_KERNEL ); +if (!default_dram_type) +panic ("%s() failed to allocate default DRAM tier\n" , __func__); +mutex_lock (&memory_tier_lock); + +/* For now let's have 4 memory tier below default DRAM tier. */ + default_dram_type->adistance =MEMTIER_ADISTANCE_DRAM ; +INIT_LIST_HEAD (&default_dram_type->tier_sibiling); + default_dram_type->memtier =NULL ; +kref_init (&default_dram_type->kref);It appears that we can define a function to initialize a memory type.
I added the below
static inline struct memory_dev_type *alloc_memory_type(int adistance) static inline void destroy_memory_type(struct memory_dev_type *memtype)
Quoted 19 lines
Quoted 6 lines
mm/memory-tiers.c +1
-2/* CPU only nodes are not part of memory tiers. */ - default_dram_type.nodes = node_states[N_MEMORY ]; + default_dram_type->nodes = node_states[N_MEMORY ]; - memtier =find_create_memory_tier (&default_dram_type);mm/memory-tiers.c static int __init memory_tier_init(void) · line 219
Quoted 10 lines
mm/memory-tiers.c +2
-1+ memtier =find_create_memory_tier (default_dram_type);if (IS_ERR (memtier))panic ("%s() failed to register memory tier: %ld\n" , __func__,PTR_ERR (memtier));for_each_node_state (node,N_MEMORY ) - node_memory_types[node] = &default_dram_type; +__init_node_memory_type (node, default_dram_type);mutex_unlock (&memory_tier_lock);Best Regards,
Huang, Ying
Re: [PATCH v13 4/9] mm/demotion/dax/kmem: Set node's abstract distance to MEMTIER_DEFAULT_DAX_ADISTANCE
Quoted 19 lines · Aneesh Kumar K V
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K.V
By default, all nodes are assigned to the default memory tier which is the memory tier designated for nodes with DRAM
Set dax kmem device node's tier to slower memory tier by assigning abstract distance to
MEMTIER_DEFAULT_DAX_ADISTANCE. Low-level drivers likepapr_scmor ACPI NFIT can initialize memory device type to a more accurate value based on device tree details or HMAT.Quoted 7 lines
I don't know how ACPI NFIT can help here. Can you teach me?
Per my understanding, we may use the information provided by ACPI SLIT or HMAT (or device tree via
papr_scm) to create memory types. Before that is implemented, we just create a memory type with default abstract distance.
Quoted 6 lines
My idea is to use ACPI NFIT driver that creates a persistent memory region (
nvdimm_region_create) to also create memory type and assign that to the NUMA node mapping that region. For now NFIT driver manages all the persistent memory region/DIMM creation (drivers/acpi/nfit/core.c). It can also do the memory type creation using other ACPI information like SLIT/HMAT etc. Similarly CXL driver can do the same using CDAT.
I still think that it's better to create memory types in dax/kmem.c and CXL driver. But we can discuss further on that later.
Best Regards,
Huang, Ying
PATCH v13 5/9 mm/demotion: Build demotion targets based on explicit memory tiers
In reply to Aneesh Kumar K.VThis patch switch the demotion target building logic to use memory tiers instead of NUMA distance. All N_MEMORY NUMA nodes will be placed in the default memory tier and additional memory tiers will be added by drivers like dax kmem.
This patch builds the demotion target for a NUMA node by looking at all memory tiers below the tier to which the NUMA node belongs. The closest node in the immediately following memory tier is used as a demotion target.
Since we are now only building demotion target for N_MEMORY NUMA nodes the CPU hotplug calls are removed in this patch.
--- include/linux/memory-tiers.h | 13 ++ include/linux/migrate.h | 13 -- mm/memory-tiers.c | 219 ++++++++++++++++++- mm/migrate.c | 394 ----------------------------------- mm/vmstat.c | 4 - 5 files changed, 229 insertions(+), 414 deletions(-)
include/linux/memory-tiers.h +13 -0
@@ -31,6 +31,14 @@ struct memory_dev_type {extern bool numa_demotion_enabled;void init_node_memory_type (int node,struct memory_dev_type *default_type);void clear_node_memory_type (int node,struct memory_dev_type *memtype); +#ifdef CONFIG_MIGRATION +int next_demotion_node (int node); +#else +static inline int next_demotion_node (int node) +{ +return NUMA_NO_NODE ; +} +#endif #else
@@ -44,5 +52,10 @@ static inline void clear_node_memory_type(int node, struct memory_dev_type *memt { } + +static inline int next_demotion_node (int node) +{ +return NUMA_NO_NODE ; +}#endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */
include/linux/migrate.h +0 -13
@@ -100,19 +100,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ -#if defined(CONFIG_MIGRATION ) && defined(CONFIG_NUMA ) -extern void set_migration_target_nodes (void ); -extern void migrate_on_reclaim_init (void ); -extern int next_demotion_node (int node); -#else -static inline void set_migration_target_nodes (void ) {} -static inline void migrate_on_reclaim_init (void ) {} -static inline int next_demotion_node (int node) -{ -return NUMA_NO_NODE ; -} -#endif -#ifdef CONFIG_COMPACTION bool PageMovable (struct page *page);void __SetPageMovable (struct page *page,const struct movable_operations *ops);
mm/memory-tiers.c +216 -3
265 lines not on this page · raw message
mm/migrate.c +0 -394
398 lines not on this page · raw message
mm/vmstat.c +0 -4
@@ -28,7 +28,6 @@#include <linux/mm_inline.h> #include <linux/page_ext.h> #include <linux/page_owner.h> -#include <linux/migrate.h> #include "internal.h"
@@ -2060,7 +2059,6 @@ static int vmstat_cpu_online(unsigned int cpu)if (!node_state (cpu_to_node (cpu),N_CPU )) {node_set_state (cpu_to_node (cpu),N_CPU ); -set_migration_target_nodes (); }return 0 ;
@@ -2085,7 +2083,6 @@ static int vmstat_cpu_dead(unsigned int cpu)return 0 ;node_clear_state (node,N_CPU ); -set_migration_target_nodes ();return 0 ; }
@@ -2118,7 +2115,6 @@ void __init init_mm_internals(void)start_shepherd_timer ();#endif -migrate_on_reclaim_init ();#ifdef CONFIG_PROC_FS proc_create_seq ("buddyinfo" ,0444 ,NULL , &fragmentation_op);proc_create_seq ("pagetypeinfo" ,0400 ,NULL , &pagetypeinfo_op);
Signature · 1 line
2.37.1
PATCH v13 6/9 mm/demotion: Add pg_data_t member to track node memory tier details
In reply to Aneesh Kumar K.VAlso update different helpes to use NODE_DATA()->memtier. Since node specific memtier can change based on the reassignment of NUMA node to a different memory tiers, accessing NODE_DATA()->memtier needs to happen under an rcu read lock or memory_tier_lock.
--- include/linux/mmzone.h | 3 +++ mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-)
include/linux/mmzone.h +3 -0
@@ -928,6 +928,9 @@ typedef struct pglist_data {/* Per-node vmstats */ struct per_cpu_nodestat __percpu *per_cpu_nodestats;atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS ]; +#ifdef CONFIG_NUMA +struct memory_tier __rcu *memtier; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)
mm/memory-tiers.c +32 -6
@@ -5,6 +5,7 @@#include <linux/kobject.h> #include <linux/memory.h> #include <linux/random.h> +#include <linux/mmzone.h> #include <linux/memory-tiers.h> #include "internal.h"
@@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtystatic struct memory_tier *__node_get_memory_tier (int node) { -struct memory_dev_type *memtype; +pg_data_t *pgdat; - memtype = node_memory_types[node]; -if (memtype &&node_isset (node, memtype->nodes)) -return memtype->memtier; -return NULL ; + pgdat =NODE_DATA (node); +if (!pgdat) +return NULL ; +/* +* Since we hold memory_tier_lock, we can avoid +* RCU read locks when accessing the details. No +* parallel updates are possible here. +*/ +return rcu_dereference_check (pgdat->memtier, +lockdep_is_held (&memory_tier_lock)); }#ifdef CONFIG_MIGRATION
@@ -295,6 +302,8 @@ static struct memory_tier *set_node_memory_tier(int node) {struct memory_tier *memtier;struct memory_dev_type *memtype; +pg_data_t *pgdat =NODE_DATA (node); +lockdep_assert_held_once (&memory_tier_lock);
@@ -307,6 +316,8 @@ static struct memory_tier *set_node_memory_tier(int node) memtype = node_memory_types[node];node_set (node, memtype->nodes); memtier =find_create_memory_tier (memtype); +if (!IS_ERR (memtier)) +rcu_assign_pointer (pgdat->memtier, memtier);return memtier; }
@@ -319,12 +330,25 @@ static void destroy_memory_tier(struct memory_tier *memtier)static bool clear_node_memory_tier (int node) {bool cleared = false; +pg_data_t *pgdat;struct memory_tier *memtier; + pgdat =NODE_DATA (node); +if (!pgdat) +return false; + +/* +* Make sure that anybody looking at NODE_DATA who finds +* a valid memtier finds memory_dev_types with nodes still +* linked to the memtier. We achieve this by waiting for +* rcu read section to finish using synchronize_rcu. +*/ memtier =__node_get_memory_tier (node);if (memtier) {struct memory_dev_type *memtype; +rcu_assign_pointer (pgdat->memtier,NULL ); +synchronize_rcu (); memtype = node_memory_types[node];node_clear (node, memtype->nodes);if (nodes_empty (memtype->nodes)) {
@@ -422,8 +446,10 @@ static int __init memory_tier_init(void)panic ("%s() failed to register memory tier: %ld\n" , __func__,PTR_ERR (memtier)); -for_each_node_state (node,N_MEMORY ) +for_each_node_state (node,N_MEMORY ) {__init_node_memory_type (node, default_dram_type); +rcu_assign_pointer (NODE_DATA (node)->memtier, memtier); + }mutex_unlock (&memory_tier_lock);#ifdef CONFIG_MIGRATION
Signature · 1 line
2.37.1
Re: [PATCH v13 6/9] mm/demotion: Add pg_data_t member to track node memory tier details
Quoted 86 lines · Aneesh Kumar K.V
Also update different helpes to use NODE_DATA()->memtier. Since node specific memtier can change based on the reassignment of NUMA node to a different memory tiers, accessing NODE_DATA()->memtier needs to happen under an rcu read lock or
memory_tier_lock.Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/mmzone.h | 3 +++ mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-)include/linux/mmzone.h +3
-0@@ -928,6 +928,9 @@ typedef struct pglist_data {/* Per-node vmstats */ struct per_cpu_nodestat __percpu *per_cpu_nodestats;atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS ]; +#ifdef CONFIG_NUMA +struct memory_tier __rcu *memtier; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)mm/memory-tiers.c +17
-5@@ -5,6 +5,7 @@#include <linux/kobject.h> #include <linux/memory.h> #include <linux/random.h> +#include <linux/mmzone.h> #include <linux/memory-tiers.h> #include "internal.h" @@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtystatic struct memory_tier *__node_get_memory_tier (int node) { -struct memory_dev_type *memtype; +pg_data_t *pgdat; - memtype = node_memory_types[node]; -if (memtype &&node_isset (node, memtype->nodes)) -return memtype->memtier; -return NULL ; + pgdat =NODE_DATA (node); +if (!pgdat) +return NULL ; +/* +* Since we hold memory_tier_lock, we can avoid +* RCU read locks when accessing the details. No +* parallel updates are possible here. +*/ +return rcu_dereference_check (pgdat->memtier, +lockdep_is_held (&memory_tier_lock)); }#ifdef CONFIG_MIGRATION @@ -295,6 +302,8 @@ static struct memory_tier *set_node_memory_tier(int node) {struct memory_tier *memtier;struct memory_dev_type *memtype; +pg_data_t *pgdat =NODE_DATA (node); +lockdep_assert_held_once (&memory_tier_lock);@@ -307,6 +316,8 @@ static struct memory_tier *set_node_memory_tier(int node) memtype = node_memory_types[node];node_set (node, memtype->nodes); memtier =find_create_memory_tier (memtype); +if (!IS_ERR (memtier)) +rcu_assign_pointer (pgdat->memtier, memtier);return memtier; }@@ -319,12 +330,25 @@ static void destroy_memory_tier(struct memory_tier *memtier)static bool clear_node_memory_tier (int node) {bool cleared = false; +pg_data_t *pgdat;struct memory_tier *memtier;
mm/memory-tiers.c static void destroy_memory_tier(struct memory_tier *memtier) · line 345
Quoted 10 lines
mm/memory-tiers.c +10
-0+ pgdat =NODE_DATA (node); +if (!pgdat) +return false; + +/* +* Make sure that anybody looking at NODE_DATA who finds +* a valid memtier finds memory_dev_types with nodes still +* linked to the memtier. We achieve this by waiting for +* rcu read section to finish using synchronize_rcu. +*/
The synchronize_rcu() is also needed because we may free the memory tier.
Best Regards,
Huang, Ying
Quoted 21 lines
mm/memory-tiers.c +5
-1memtier =__node_get_memory_tier (node);if (memtier) {struct memory_dev_type *memtype; +rcu_assign_pointer (pgdat->memtier,NULL ); +synchronize_rcu (); memtype = node_memory_types[node];node_clear (node, memtype->nodes);if (nodes_empty (memtype->nodes)) {@@ -422,8 +446,10 @@ static int __init memory_tier_init(void)panic ("%s() failed to register memory tier: %ld\n" , __func__,PTR_ERR (memtier)); -for_each_node_state (node,N_MEMORY ) +for_each_node_state (node,N_MEMORY ) {__init_node_memory_type (node, default_dram_type); +rcu_assign_pointer (NODE_DATA (node)->memtier, memtier); + }mutex_unlock (&memory_tier_lock);#ifdef CONFIG_MIGRATION
Re: [PATCH v13 6/9] mm/demotion: Add pg_data_t member to track node memory tier details
In reply to Aneesh Kumar K.VQuoted 38 lines · Aneesh Kumar K.V
Also update different helpes to use NODE_DATA()->memtier. Since node specific memtier can change based on the reassignment of NUMA node to a different memory tiers, accessing NODE_DATA()->memtier needs to happen under an rcu read lock or
memory_tier_lock.Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/mmzone.h | 3 +++ mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-)include/linux/mmzone.h +3
-0@@ -928,6 +928,9 @@ typedef struct pglist_data {/* Per-node vmstats */ struct per_cpu_nodestat __percpu *per_cpu_nodestats;atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS ]; +#ifdef CONFIG_NUMA +struct memory_tier __rcu *memtier; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)mm/memory-tiers.c +1
-0@@ -5,6 +5,7 @@#include <linux/kobject.h> #include <linux/memory.h> #include <linux/random.h> +#include <linux/mmzone.h> #include <linux/memory-tiers.h> #include "internal.h"
mm/memory-tiers.c static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · line 145
Quoted 10 lines
mm/memory-tiers.c +1
-5@@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtystatic struct memory_tier *__node_get_memory_tier (int node) { -struct memory_dev_type *memtype; +pg_data_t *pgdat; - memtype = node_memory_types[node]; -if (memtype &&node_isset (node, memtype->nodes)) -return memtype->memtier; -return NULL ;
After adding pgdat->memtier, it appears there's unnecessary to keep memtype->memtier?
Best Regards,
Huang, Ying
Quoted 69 lines
mm/memory-tiers.c +30
-1+ pgdat =NODE_DATA (node); +if (!pgdat) +return NULL ; +/* +* Since we hold memory_tier_lock, we can avoid +* RCU read locks when accessing the details. No +* parallel updates are possible here. +*/ +return rcu_dereference_check (pgdat->memtier, +lockdep_is_held (&memory_tier_lock)); }#ifdef CONFIG_MIGRATION @@ -295,6 +302,8 @@ static struct memory_tier *set_node_memory_tier(int node) {struct memory_tier *memtier;struct memory_dev_type *memtype; +pg_data_t *pgdat =NODE_DATA (node); +lockdep_assert_held_once (&memory_tier_lock);@@ -307,6 +316,8 @@ static struct memory_tier *set_node_memory_tier(int node) memtype = node_memory_types[node];node_set (node, memtype->nodes); memtier =find_create_memory_tier (memtype); +if (!IS_ERR (memtier)) +rcu_assign_pointer (pgdat->memtier, memtier);return memtier; }@@ -319,12 +330,25 @@ static void destroy_memory_tier(struct memory_tier *memtier)static bool clear_node_memory_tier (int node) {bool cleared = false; +pg_data_t *pgdat;struct memory_tier *memtier; + pgdat =NODE_DATA (node); +if (!pgdat) +return false; + +/* +* Make sure that anybody looking at NODE_DATA who finds +* a valid memtier finds memory_dev_types with nodes still +* linked to the memtier. We achieve this by waiting for +* rcu read section to finish using synchronize_rcu. +*/ memtier =__node_get_memory_tier (node);if (memtier) {struct memory_dev_type *memtype; +rcu_assign_pointer (pgdat->memtier,NULL ); +synchronize_rcu (); memtype = node_memory_types[node];node_clear (node, memtype->nodes);if (nodes_empty (memtype->nodes)) {@@ -422,8 +446,10 @@ static int __init memory_tier_init(void)panic ("%s() failed to register memory tier: %ld\n" , __func__,PTR_ERR (memtier)); -for_each_node_state (node,N_MEMORY ) +for_each_node_state (node,N_MEMORY ) {__init_node_memory_type (node, default_dram_type); +rcu_assign_pointer (NODE_DATA (node)->memtier, memtier); + }mutex_unlock (&memory_tier_lock);#ifdef CONFIG_MIGRATION
Re: [PATCH v13 6/9] mm/demotion: Add pg_data_t member to track node memory tier details
Quoted 46 lines · Ying
Quoted 44 lines · Aneesh Kumar K.V
Also update different helpes to use NODE_DATA()->memtier. Since node specific memtier can change based on the reassignment of NUMA node to a different memory tiers, accessing NODE_DATA()->memtier needs to happen under an rcu read lock or
memory_tier_lock.Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/mmzone.h | 3 +++ mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-)include/linux/mmzone.h +3
-0@@ -928,6 +928,9 @@ typedef struct pglist_data {/* Per-node vmstats */ struct per_cpu_nodestat __percpu *per_cpu_nodestats;atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS ]; +#ifdef CONFIG_NUMA +struct memory_tier __rcu *memtier; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)mm/memory-tiers.c +2
-1@@ -5,6 +5,7 @@#include <linux/kobject.h> #include <linux/memory.h> #include <linux/random.h> +#include <linux/mmzone.h> #include <linux/memory-tiers.h> #include "internal.h" @@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtystatic struct memory_tier *__node_get_memory_tier (int node) { -struct memory_dev_type *memtype; +pg_data_t *pgdat;
Quoted 8 lines
mm/memory-tiers.c static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · line 145
Quoted 4 lines
mm/memory-tiers.c +0
-4- memtype = node_memory_types[node]; -if (memtype &&node_isset (node, memtype->nodes)) -return memtype->memtier; -return NULL ;After adding
pgdat->memtier, it appears there's unnecessary to keepmemtype->memtier?
It do simplify find_create_memory_tier() where I use if (memtype->memtier) to check whether the memtype is already added to a memory tier. I could switch that to list_empty(memtype->tier_sibiling). But I felt the current one is much cleaner
.
-aneesh
Re: [PATCH v13 6/9] mm/demotion: Add pg_data_t member to track node memory tier details
Quoted 56 lines · Aneesh Kumar K V
Quoted 46 lines · Ying
Quoted 44 lines · Aneesh Kumar K.V
Also update different helpes to use NODE_DATA()->memtier. Since node specific memtier can change based on the reassignment of NUMA node to a different memory tiers, accessing NODE_DATA()->memtier needs to happen under an rcu read lock or
memory_tier_lock.Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/mmzone.h | 3 +++ mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-)include/linux/mmzone.h +3
-0@@ -928,6 +928,9 @@ typedef struct pglist_data {/* Per-node vmstats */ struct per_cpu_nodestat __percpu *per_cpu_nodestats;atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS ]; +#ifdef CONFIG_NUMA +struct memory_tier __rcu *memtier; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)mm/memory-tiers.c +2
-1@@ -5,6 +5,7 @@#include <linux/kobject.h> #include <linux/memory.h> #include <linux/random.h> +#include <linux/mmzone.h> #include <linux/memory-tiers.h> #include "internal.h" @@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtystatic struct memory_tier *__node_get_memory_tier (int node) { -struct memory_dev_type *memtype; +pg_data_t *pgdat;Quoted 8 lines
mm/memory-tiers.c static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · line 145
Quoted 4 lines
mm/memory-tiers.c +0
-4- memtype = node_memory_types[node]; -if (memtype &&node_isset (node, memtype->nodes)) -return memtype->memtier; -return NULL ;After adding
pgdat->memtier, it appears there's unnecessary to keepmemtype->memtier?
Quoted 4 lines
It do simplify
find_create_memory_tier()where I use if (memtype->memtier) to check whether the memtype is already added to a memory tier. I could switch that tolist_empty(memtype->tier_sibiling). But I felt the current one is much cleaner
I prefer "list_empty(memtype->tier_sibiling)". But I will let you to decide.
Best Regards,
Huang, Ying
PATCH v13 7/9 mm/demotion: Demote pages according to allocation fallback order
In reply to Aneesh Kumar K.VFrom: Jagdish Gediya <[email protected]>
Currently, a higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path. This strict demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space). This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from any lower tier, whereas the demotion order doesn't allow that currently.
This patch adds support to get all the allowed demotion targets for a memory tier. demote_page_list() function is now modified to utilize this allowed node mask as the fallback allocation mask.
--- include/linux/memory-tiers.h | 12 ++++++++ mm/memory-tiers.c | 51 +++++++++++++++++++++++++++++-- mm/vmscan.c | 58 ++++++++++++++++++++++++++---------- 3 files changed, 103 insertions(+), 18 deletions(-)
include/linux/memory-tiers.h +12 -0
@@ -4,6 +4,7 @@#include <linux/types.h> #include <linux/nodemask.h> +#include <linux/mmzone.h> /* * Each tier cover a abstrace distance chunk size of 128 */
@@ -33,11 +34,17 @@ void init_node_memory_type(int node, struct memory_dev_type *default_type);void clear_node_memory_type (int node,struct memory_dev_type *memtype);#ifdef CONFIG_MIGRATION int next_demotion_node (int node); +void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets);#else static inline int next_demotion_node (int node) {return NUMA_NO_NODE ; } + +static inline void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets) +{ + *targets =NODE_MASK_NONE ; +}#endif #else
@@ -57,5 +64,10 @@ static inline int next_demotion_node(int node) {return NUMA_NO_NODE ; } + +static inline void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets) +{ + *targets =NODE_MASK_NONE ; +}#endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */
mm/memory-tiers.c +48 -3
90 lines not on this page · raw message
mm/vmscan.c +43 -15
80 lines not on this page · raw message
Signature · 1 line
2.37.1
PATCH v13 8/9 mm/demotion: Update node_is_toptier to work with memory tiers
In reply to Aneesh Kumar K.VWith memory tiers support we can have memory only NUMA nodes in the top tier from which we want to avoid promotion tracking NUMA faults. Update node_is_toptier to work with memory tiers. All NUMA nodes are by default top tier nodes. With lower memory tiers added we consider all memory tiers above a memory tier having CPU NUMA nodes as a top memory tier
--- include/linux/memory-tiers.h | 11 +++++++++ include/linux/node.h | 5 ---- mm/huge_memory.c | 1 + mm/memory-tiers.c | 46 ++++++++++++++++++++++++++++++++++++ mm/migrate.c | 1 + mm/mprotect.c | 1 + 6 files changed, 60 insertions(+), 5 deletions(-)
include/linux/memory-tiers.h +11 -0
@@ -35,6 +35,7 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype);#ifdef CONFIG_MIGRATION int next_demotion_node (int node);void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets); +bool node_is_toptier (int node);#else static inline int next_demotion_node (int node) {
@@ -45,6 +46,11 @@ static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *target { *targets =NODE_MASK_NONE ; } + +static inline bool node_is_toptier (int node) +{ +return true; +}#endif #else
@@ -69,5 +75,10 @@ static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *target { *targets =NODE_MASK_NONE ; } + +static inline bool node_is_toptier (int node) +{ +return true; +}#endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */
include/linux/node.h +0 -5
@@ -185,9 +185,4 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,#define to_node (device) container_of(device, struct node, dev) -static inline bool node_is_toptier (int node) -{ -return node_state (node,N_CPU ); -} -#endif /* _LINUX_NODE_H_ */
mm/huge_memory.c +1 -0
@@ -36,6 +36,7 @@#include <linux/numa.h> #include <linux/page_owner.h> #include <linux/sched/sysctl.h> +#include <linux/memory-tiers.h> #include <asm/tlb.h> #include <asm/pgalloc.h>
mm/memory-tiers.c +46 -0
64 lines not on this page · raw message
mm/migrate.c +1 -0
@@ -50,6 +50,7 @@#include <linux/memory.h> #include <linux/random.h> #include <linux/sched/sysctl.h> +#include <linux/memory-tiers.h> #include <asm/tlbflush.h>
mm/mprotect.c +1 -0
@@ -31,6 +31,7 @@#include <linux/pgtable.h> #include <linux/sched/sysctl.h> #include <linux/userfaultfd_k.h> +#include <linux/memory-tiers.h> #include <asm/cacheflush.h> #include <asm/mmu_context.h> #include <asm/tlbflush.h>
Signature · 1 line
2.37.1
PATCH v13 9/9 lib/nodemask: Optimize node_random for nodemask with single NUMA node
In reply to Aneesh Kumar K.VThe most common case for certain node_random usage (demotion nodemask) is with nodemask weight 1. We can avoid calling get_random_init() in that case and always return the only node set in the nodemask.
--- lib/nodemask.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
lib/nodemask.c +12 -3
@@ -20,12 +20,21 @@ EXPORT_SYMBOL(__next_node_in);*/ int node_random (const nodemask_t *maskp) { -int w, bit =NUMA_NO_NODE ; +int w, bit; w =nodes_weight (*maskp); -if (w) +switch (w) { +case 0 : + bit =NUMA_NO_NODE ; +break ; +case 1 : + bit =__first_node (maskp); +break ; +default : bit =bitmap_ord_to_pos (maskp->bits, -get_random_int () % w,MAX_NUMNODES ); +get_random_int () % w,MAX_NUMNODES ); +break ; + }return bit; }#endif
Signature · 1 line
2.37.1
Re: [PATCH v13 9/9] lib/nodemask: Optimize node_random for nodemask with single NUMA node
Quoted 3 lines · Aneesh Kumar K.V
The most common case for certain
node_randomusage (demotion nodemask) is with nodemask weight 1. We can avoid callingget_random_init()in that case and always return the only node set in the nodemask.
I think that this patch can sit between [5/9] and [6/9], just after it is used.
Quoted 14 lines
Signed-off-by Aneesh Kumar K.V <[email protected]>--- lib/nodemask.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)lib/nodemask.c +0
-1@@ -20,12 +20,21 @@ EXPORT_SYMBOL(__next_node_in);*/ int node_random (const nodemask_t *maskp) { -int w, bit =NUMA_NO_NODE ;
lib/nodemask.c EXPORT_SYMBOL(__next_node_in); · line 31
Quoted 10 lines
lib/nodemask.c +7
-1+int w, bit; w =nodes_weight (*maskp); -if (w) +switch (w) { +case 0 : + bit =NUMA_NO_NODE ; +break ; +case 1 : + bit =__first_node (maskp);
Per my understanding, first_node() is the formal API and we should use that? Just like we use nodes_weight() instead of __nodes_weight().
Best Regards,
Huang, Ying
Quoted 10 lines
lib/nodemask.c +5
-1+break ; +default : bit =bitmap_ord_to_pos (maskp->bits, -get_random_int () % w,MAX_NUMNODES ); +get_random_int () % w,MAX_NUMNODES ); + break; + }return bit; }#endif
Re: [PATCH v13 9/9] lib/nodemask: Optimize node_random for nodemask with single NUMA node
Quoted 25 lines · Ying
Quoted 3 lines · Aneesh Kumar K.V
The most common case for certain
node_randomusage (demotion nodemask) is with nodemask weight 1. We can avoid callingget_random_init()in that case and always return the only node set in the nodemask.I think that this patch can sit between [5/9] and [6/9], just after it is used.
Quoted 16 lines
Signed-off-by Aneesh Kumar K.V <[email protected]>--- lib/nodemask.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)lib/nodemask.c +1
-1@@ -20,12 +20,21 @@ EXPORT_SYMBOL(__next_node_in);*/ int node_random (const nodemask_t *maskp) { -int w, bit =NUMA_NO_NODE ; +int w, bit;
Quoted 12 lines
lib/nodemask.c EXPORT_SYMBOL(__next_node_in); · line 31
Quoted 8 lines
lib/nodemask.c +6
-1w =nodes_weight (*maskp); -if (w) +switch (w) { +case 0 : + bit =NUMA_NO_NODE ; +break ; +case 1 : + bit =__first_node (maskp);Per my understanding,
first_node()is the formal API and we should use that? Just like we usenodes_weight()instead of__nodes_weight().
updated.
-aneesh