mm/demotion: Memory tiers and demotion
Subsystems (12)
v6 cover mm/demotion: Memory tiers and demotion
The current kernel has the basic memory tiering support: Inactive pages on a higher tier NUMA node can be migrated (demoted) to a lower tier NUMA node to make room for new allocations on the higher tier NUMA node. Frequently accessed pages on a lower tier NUMA node can be migrated (promoted) to a higher tier NUMA node to improve the performance.
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel interface needs to be improved for several important use cases:
* The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.
* The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM (e.g. GPU memory) devices, these memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
* Also because the current tier hierarchy always puts CPU nodes into the top tier, when a CPU is hot-added (or hot-removed) and triggers a memory node from CPU-less into a CPU node (or vice versa), the memory tier hierarchy gets changed, even though no memory node is added or removed. This can make the tier hierarchy unstable and make it difficult to support tier-based memory accounting.
* A higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), and has resulted in the feature request for an interface to override the system-wide, per-node demotion order from the userspace. This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from any lower tier, whereas the demotion order doesn't allow that.
* There are no interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.
This patch series make the creation of memory tiers explicit under the control of userspace or device driver.
Memory Tier Initialization
==========================
By default, all memory nodes are assigned to the default tier (1). The default tier device has a rank value (200).
A device driver can move up or down its memory nodes from the default tier. For example, PMEM can move down its memory nodes below the default tier, whereas GPU can move up its memory nodes above the default tier.
The kernel initialization code makes the decision on which exact tier a memory node should be assigned to based on the requests from the device drivers as well as the memory device hardware information provided by the firmware.
Hot-adding/removing CPUs doesn't affect memory tier hierarchy.
Memory Allocation for Demotion
==============================
This patch series keep the demotion target page allocation logic same. The demotion page allocation pick the closest NUMA node in the next lower tier to the current NUMA node allocating pages from.
This will be later improved to use the same page allocation strategy using fallback list.
Sysfs Interface:
=======================
Listing current list of memory tiers and rank details:
:/sys/devices/system/memtier$ ls default_tier max_tier memtier1 power uevent :/sys/devices/system/memtier$ cat default_tier memtier1 :/sys/devices/system/memtier$ cat max_tier 3 :/sys/devices/system/memtier$
Per node memory tier details:
For a cpu only NUMA node:
:/sys/devices/system/node# cat node0/memtier :/sys/devices/system/node# echo 1 > node0/memtier :/sys/devices/system/node# cat node0/memtier :/sys/devices/system/node#
For a NUMA node with memory:
:/sys/devices/system/node# cat node1/memtier 1 :/sys/devices/system/node# ls ../memtier/ default_tier max_tier memtier1 power uevent :/sys/devices/system/node# echo 2 > node1/memtier :/sys/devices/system/node# :/sys/devices/system/node# ls ../memtier/ default_tier max_tier memtier1 memtier2 power uevent :/sys/devices/system/node# cat node1/memtier 2 :/sys/devices/system/node# :/sys/devices/system/node# cat ../memtier/memtier2/rank 100 :/sys/devices/system/node# :/sys/devices/system/node# cat ../memtier/memtier1/rank 200 :/sys/devices/system/node#
Removing a NUMA node from demotion:
:/sys/devices/system/node# cat node1/memtier 2 :/sys/devices/system/node# echo none > node1/memtier :/sys/devices/system/node# :/sys/devices/system/node# cat node1/memtier :/sys/devices/system/node# :/sys/devices/system/node# ls ../memtier/ default_tier max_tier memtier1 power uevent :/sys/devices/system/node#
The above also resulted in removal of memtier2 which was created in the earlier step.
Changes from v5:
* Remove patch supporting N_MEMORY node removal from memory tiers. memory tiers are going to be used for features other than demotion. Hence keep all N_MEMORY nodes in memory tiers irrespective of whether they want to participate in promotion or demotion.
* Add NODE_DATA->memtier
* Rearrage patches to add sysfs files later.
* Add support to create memory tiers from userspace.
* Address other review feedback.
Changes from v4:
* Address review feedback.
* Reverse the meaning of "rank": higher rank value means higher tier.
* Add "/sys/devices/system/memtier/default_tier".
* Add node_is_toptier
v4:
Add support for explicit memory tiers and ranks.
v3:
- Modify patch 1 subject to make it more specific
- Remove /sys/kernel/mm/numa/demotion_targets interface, use /sys/devices/system/node/demotion_targets instead and make it writable to override node_states[N_DEMOTION_TARGETS].
- Add support to view per node demotion targets via sysfs
v2:
In v1, only 1st patch of this patch series was sent, which was implemented to avoid some of the limitations on the demotion target sharing, however for certain numa topology, the demotion targets found by that patch was not most optimal, so 1st patch in this series is modified according to suggestions from Huang and Baolin. Different examples of demotion list comparasion between existing implementation and changed implementation can be found in the commit message of 1st patch.
Aneesh Kumar K.V (11):
mm/demotion: Add support for explicit memory tiers mm/demotion: Move memory demotion related code mm/demotion: Return error on write to numa_demotion sysfs mm/demotion/dax/kmem: Set node's memory tier to MEMORY_TIER_PMEM mm/demotion: Build demotion targets based on explicit memory tiers mm/demotion: Expose memory tier details via sysfs mm/demotion: Add per node memory tier attribute to sysfs mm/demotion: Add support for memory tier creation from userspace mm/demotion: Add pg_data_t member to track node memory tier details mm/demotion: Update node_is_toptier to work with memory tiers mm/demotion: Add sysfs ABI documentation
Jagdish Gediya (2):
mm/demotion: Demote pages according to allocation fallback order mm/demotion: Add documentation for memory tiering
.../ABI/testing/sysfs-kernel-mm-memory-tiers | 87 ++ Documentation/admin-guide/mm/index.rst | 1 + .../admin-guide/mm/memory-tiering.rst | 181 ++++ drivers/base/node.c | 39 + drivers/dax/kmem.c | 4 + include/linux/memory-tiers.h | 63 ++ include/linux/migrate.h | 15 - include/linux/mmzone.h | 3 + include/linux/node.h | 5 - mm/Kconfig | 3 + mm/Makefile | 1 + mm/huge_memory.c | 1 + mm/memory-tiers.c | 888 ++++++++++++++++++ mm/migrate.c | 453 +-------- mm/mprotect.c | 1 + mm/vmscan.c | 57 +- mm/vmstat.c | 4 - 17 files changed, 1316 insertions(+), 490 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers create mode 100644 Documentation/admin-guide/mm/memory-tiering.rst create mode 100644 include/linux/memory-tiers.h create mode 100644 mm/memory-tiers.c
Signature · 1 line
2.36.1
PATCH v6 1/13 mm/demotion: Add support for explicit memory tiers
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel interface needs to be improved for several important use cases,
The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.
The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.
This patch series address the above by defining memory tiers explicitly.
This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at
"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.
For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier and memtier2 is the lowest tier.
The rank value of each memtier should be unique.
A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.
This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2) which are created by different kernel subsystems. The default memory tier created by the kernel is memtier1. Once created these memory tiers are not destroyed even if they don't have any NUMA nodes assigned to them.
This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
/sys/devices/system/memtier/memtierN/
The nodes which are part of a specific memory tier can be listed via/sys/devices/system/memtier/memtierN/nodelist
--- include/linux/memory-tiers.h | 20 ++++++++ mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++ 4 files changed, 113 insertions(+) create mode 100644 include/linux/memory-tiers.h create mode 100644 mm/memory-tiers.c
include/linux/memory-tiers.h new file +20 -0
@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_TIERS_H +#define _LINUX_MEMORY_TIERS_H + +#ifdef CONFIG_TIERED_MEMORY + +#define MEMORY_TIER_HBM_GPU 0 +#define MEMORY_TIER_DRAM 1 +#define MEMORY_TIER_PMEM 2 + +#define MEMORY_RANK_HBM_GPU 300 +#define MEMORY_RANK_DRAM 200 +#define MEMORY_RANK_PMEM 100 + +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM +#define MAX_MEMORY_TIERS 3 + +#endif /* CONFIG_TIERED_MEMORY */ + +#endif
mm/Kconfig +3 -0
@@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION config ARCH_ENABLE_THP_MIGRATION bool +config TIERED_MEMORY + def_bool NUMA + config HUGETLB_PAGE_SIZE_VARIABLE def_bool n help
mm/Makefile +1 -0
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
mm/memory-tiers.c new file +89 -0
89 lines not on this page · raw message
Signature · 1 line
2.36.1
Re: [PATCH v6 01/13] mm/demotion: Add support for explicit memory tiers
Hi, Aneesh,
Quoted 123 lines · Aneesh Kumar K.V
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel interface needs to be improved for several important use cases,
The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.
This patch series address the above by defining memory tiers explicitly.
This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at
"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.
For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier and memtier2 is the lowest tier.The rank value of each memtier should be unique.
A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.
This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2) which are created by different kernel subsystems. The default memory tier created by the kernel is memtier1. Once created these memory tiers are not destroyed even if they don't have any NUMA nodes assigned to them.
This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
/sys/devices/system/memtier/memtierN/The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelistSuggested-by Wei Xu <[email protected]>Signed-off-by Jagdish Gediya <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>---
include/linux/memory-tiers.h | 20 ++++++++ mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++ 4 files changed, 113 insertions(+)create mode 100644
include/linux/memory-tiers.h
create mode 100644mm/memory-tiers.cinclude/linux/memory-tiers.h new file +20
-0@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_TIERS_H +#define _LINUX_MEMORY_TIERS_H + +#ifdef CONFIG_TIERED_MEMORY + +#define MEMORY_TIER_HBM_GPU 0 +#define MEMORY_TIER_DRAM 1 +#define MEMORY_TIER_PMEM 2 + +#define MEMORY_RANK_HBM_GPU 300 +#define MEMORY_RANK_DRAM 200 +#define MEMORY_RANK_PMEM 100 + +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM +#define MAX_MEMORY_TIERS 3 + +#endif /* CONFIG_TIERED_MEMORY */ + +#endif mm/Kconfig +0
-0@@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATIONconfig
ARCH_ENABLE_THP_MIGRATION
bool
mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 619
Quoted 3 lines
mm/Kconfig +3
-0+config TIERED_MEMORY + def_bool NUMA +
As Yang pointed out, why not just use CONFIG_NUMA? I suspect the added value of CONIFIG_TIRED_MEMORY.
Quoted 51 lines
config
HUGETLB_PAGE_SIZE_VARIABLE
def_booln
helpmm/Makefile +0
-0@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/obj-$(CONFIG_FAILSLAB) += failslab.o
obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.omm/memory-tiers.c new file +30
-0@@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/types.h> +#include <linux/nodemask.h> +#include <linux/slab.h> +#include <linux/memory-tiers.h> + +struct memory_tier { +struct list_head list; +nodemask_t nodelist; +int id; +int rank; +}; + +static DEFINE_MUTEX (memory_tier_lock); +static LIST_HEAD (memory_tiers); + +/* +* Keep it simple by having direct mapping between +* tier index and rank value. +*/ +static inline int get_rank_from_tier (unsigned int tier) +{ +switch (tier) { +case MEMORY_TIER_HBM_GPU : +return MEMORY_RANK_HBM_GPU ; +case MEMORY_TIER_DRAM : +return MEMORY_RANK_DRAM ; +case MEMORY_TIER_PMEM : +return MEMORY_RANK_PMEM ; + }
mm/memory-tiers.c line 40
Quoted 10 lines
mm/memory-tiers.c new file +10
-0+return -1 ; +} + +static void insert_memory_tier (struct memory_tier *memtier) +{ +struct list_head *ent; +struct memory_tier *tmp_memtier; + +list_for_each (ent, &memory_tiers) { + tmp_memtier = list_entry(ent, struct memory_tier, list);
list_for_each_entry() ?
mm/memory-tiers.c line 48
Quoted 9 lines
mm/memory-tiers.c new file +8
-0+if (tmp_memtier->rank < memtier->rank) { +list_add_tail (&memtier->list, ent); +return ; + } + } +list_add_tail (&memtier->list, &memory_tiers); +} +
IMHO, the locking requirements are needed here as comments to avoid confusing.
Quoted 31 lines
mm/memory-tiers.c new file +31
-0+static struct memory_tier *register_memory_tier (unsigned int tier, +unsigned int rank) +{ +struct memory_tier *memtier; + +if (tier >=MAX_MEMORY_TIERS ) +return ERR_PTR (-EINVAL ); + + memtier =kzalloc (sizeof (struct memory_tier ),GFP_KERNEL ); +if (!memtier) +return ERR_PTR (-ENOMEM ); + + memtier->id = tier; + memtier->rank = rank; + +insert_memory_tier (memtier); + +return memtier; +} + +static int __init memory_tier_init (void ) +{ +struct memory_tier *memtier; + +/* +* Register only default memory tier to hide all empty +* memory tier from sysfs. +*/ + memtier =register_memory_tier (DEFAULT_MEMORY_TIER , +get_rank_from_tier (DEFAULT_MEMORY_TIER )); +
mm/memory-tiers.c line 89
Quoted 10 lines
mm/memory-tiers.c new file +10
-0+if (IS_ERR (memtier)) +panic ("%s() failed to register memory tier: %ld\n" , + __func__,PTR_ERR (memtier)); + +/* CPU only nodes are not part of memory tiers. */ + memtier->nodelist = node_states[N_MEMORY ]; + +return 0 ; +} +subsys_initcall (memory_tier_init);
Best Regards,
Huang, Ying
Re: [PATCH v6 01/13] mm/demotion: Add support for explicit memory tiers
Quoted 126 lines · Ying Huang
Hi, Aneesh,
Quoted 123 lines · Aneesh Kumar K.V
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel interface needs to be improved for several important use cases,
The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.
This patch series address the above by defining memory tiers explicitly.
This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at
"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.
For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier and memtier2 is the lowest tier.The rank value of each memtier should be unique.
A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.
This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2) which are created by different kernel subsystems. The default memory tier created by the kernel is memtier1. Once created these memory tiers are not destroyed even if they don't have any NUMA nodes assigned to them.
This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
/sys/devices/system/memtier/memtierN/The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelistSuggested-by Wei Xu <[email protected]>Signed-off-by Jagdish Gediya <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/memory-tiers.h | 20 ++++++++ mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++ 4 files changed, 113 insertions(+)create mode 100644
include/linux/memory-tiers.h
create mode 100644mm/memory-tiers.cinclude/linux/memory-tiers.h new file +20
-0@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_TIERS_H +#define _LINUX_MEMORY_TIERS_H + +#ifdef CONFIG_TIERED_MEMORY + +#define MEMORY_TIER_HBM_GPU 0 +#define MEMORY_TIER_DRAM 1 +#define MEMORY_TIER_PMEM 2 + +#define MEMORY_RANK_HBM_GPU 300 +#define MEMORY_RANK_DRAM 200 +#define MEMORY_RANK_PMEM 100 + +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM +#define MAX_MEMORY_TIERS 3 + +#endif /* CONFIG_TIERED_MEMORY */ + +#endif mm/Kconfig +0
-0@@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION config ARCH_ENABLE_THP_MIGRATION bool
Quoted 7 lines
mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 619
Quoted 3 lines
mm/Kconfig +3
-0+config TIERED_MEMORY + def_bool NUMA +As Yang pointed out, why not just use
CONFIG_NUMA? I suspect the added value ofCONIFIG_TIRED_MEMORY.
I decided to use TIERED_MEMORY to bring more clarity. It should be same now that we have moved CONFIG_MIGRATION dependencies to runtime. IMHO having CONFIG_TIERED_MEMORY is better than using CONFIG_NUMA.
Quoted 55 lines
Quoted 55 lines
mm/Kconfig +0
-0config HUGETLB_PAGE_SIZE_VARIABLE def_bool nhelp
mm/Makefile +1
-0@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.omm/memory-tiers.c new file +34
-0@@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/types.h> +#include <linux/nodemask.h> +#include <linux/slab.h> +#include <linux/memory-tiers.h> + +struct memory_tier { +struct list_head list; +nodemask_t nodelist; +int id; +int rank; +}; + +static DEFINE_MUTEX (memory_tier_lock); +static LIST_HEAD (memory_tiers); + +/* +* Keep it simple by having direct mapping between +* tier index and rank value. +*/ +static inline int get_rank_from_tier (unsigned int tier) +{ +switch (tier) { +case MEMORY_TIER_HBM_GPU : +return MEMORY_RANK_HBM_GPU ; +case MEMORY_TIER_DRAM : +return MEMORY_RANK_DRAM ; +case MEMORY_TIER_PMEM : +return MEMORY_RANK_PMEM ; + } +return -1 ; +} + +static void insert_memory_tier (struct memory_tier *memtier)
Quoted 9 lines
mm/memory-tiers.c line 40
Quoted 6 lines
mm/memory-tiers.c new file +6
-0+{ +struct list_head *ent; +struct memory_tier *tmp_memtier; + +list_for_each (ent, &memory_tiers) { + tmp_memtier = list_entry(ent, struct memory_tier, list);
list_for_each_entry()?
ent variable is used below. Hence I won't be able to use list_for_each_entry.
Quoted 13 lines
mm/memory-tiers.c line 48
Quoted 9 lines
mm/memory-tiers.c new file +8
-0+if (tmp_memtier->rank < memtier->rank) { +list_add_tail (&memtier->list, ent); +return ; + } + } +list_add_tail (&memtier->list, &memory_tiers); +} +IMHO, the locking requirements are needed here as comments to avoid confusing.
All those functions are called with memory_tier_lock_held. Infact all list operations requires that lock held. What details do you suggest we document? I can add extra comment to the mutex itself? Adding locking details to all the functions will be duplicating the same details at multiple places?
Quoted 33 lines
Quoted 33 lines
mm/memory-tiers.c new file +33
-0+static struct memory_tier *register_memory_tier (unsigned int tier, +unsigned int rank) +{ +struct memory_tier *memtier; + +if (tier >=MAX_MEMORY_TIERS ) +return ERR_PTR (-EINVAL ); + + memtier =kzalloc (sizeof (struct memory_tier ),GFP_KERNEL ); +if (!memtier) +return ERR_PTR (-ENOMEM ); + + memtier->id = tier; + memtier->rank = rank; + +insert_memory_tier (memtier); + +return memtier; +} + +static int __init memory_tier_init (void ) +{ +struct memory_tier *memtier; + +/* +* Register only default memory tier to hide all empty +* memory tier from sysfs. +*/ + memtier =register_memory_tier (DEFAULT_MEMORY_TIER , +get_rank_from_tier (DEFAULT_MEMORY_TIER )); + +if (IS_ERR (memtier)) + panic("%s() failed to register memory tier: %ld\n",
Quoted 9 lines
mm/memory-tiers.c line 89
Quoted 8 lines
mm/memory-tiers.c new file +8
-0+ __func__, PTR_ERR(memtier)); + +/* CPU only nodes are not part of memory tiers. */ + memtier->nodelist = node_states[N_MEMORY ]; + +return 0 ; +} +subsys_initcall (memory_tier_init);
-aneesh
Re: [PATCH v6 01/13] mm/demotion: Add support for explicit memory tiers
Quoted 136 lines · Aneesh Kumar K V
Quoted 127 lines · Ying Huang
Hi, Aneesh,
Quoted 124 lines · Aneesh Kumar K.V
In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.
This current memory tier kernel interface needs to be improved for several important use cases,
The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.
With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.
This patch series address the above by defining memory tiers explicitly.
This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at
"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.
For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier and memtier2 is the lowest tier.The rank value of each memtier should be unique.
A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.
This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2) which are created by different kernel subsystems. The default memory tier created by the kernel is memtier1. Once created these memory tiers are not destroyed even if they don't have any NUMA nodes assigned to them.
This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
/sys/devices/system/memtier/memtierN/The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelistSuggested-by Wei Xu <[email protected]>Signed-off-by Jagdish Gediya <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>---
include/linux/memory-tiers.h | 20 ++++++++ mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++ 4 files changed, 113 insertions(+)create mode 100644
include/linux/memory-tiers.h
create mode 100644mm/memory-tiers.cinclude/linux/memory-tiers.h new file +20
-0@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_TIERS_H +#define _LINUX_MEMORY_TIERS_H + +#ifdef CONFIG_TIERED_MEMORY + +#define MEMORY_TIER_HBM_GPU 0 +#define MEMORY_TIER_DRAM 1 +#define MEMORY_TIER_PMEM 2 + +#define MEMORY_RANK_HBM_GPU 300 +#define MEMORY_RANK_DRAM 200 +#define MEMORY_RANK_PMEM 100 + +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM +#define MAX_MEMORY_TIERS 3 + +#endif /* CONFIG_TIERED_MEMORY */ + +#endif mm/Kconfig +0
-0@@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATIONconfig
ARCH_ENABLE_THP_MIGRATION
boolQuoted 7 lines
mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 619
Quoted 3 lines
mm/Kconfig +3
-0+config TIERED_MEMORY + def_bool NUMA +As Yang pointed out, why not just use
CONFIG_NUMA? I suspect the added value ofCONIFIG_TIRED_MEMORY.
Quoted 3 lines
I decided to use
TIERED_MEMORYto bring more clarity. It should be same now that we have movedCONFIG_MIGRATIONdependencies to runtime. IMHO havingCONFIG_TIERED_MEMORYis better than usingCONFIG_NUMA.
I don't think CONFIG_TIERED_MEMORY bring no much value. It's better to use CONFIG_NUMA directly. But this is just my opinion.
Quoted 65 lines
Quoted 55 lines
Quoted 55 lines
config
HUGETLB_PAGE_SIZE_VARIABLE
def_booln
helpmm/Makefile +0
-0@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/obj-$(CONFIG_FAILSLAB) += failslab.o
obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.omm/memory-tiers.c new file +34
-0@@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/types.h> +#include <linux/nodemask.h> +#include <linux/slab.h> +#include <linux/memory-tiers.h> + +struct memory_tier { +struct list_head list; +nodemask_t nodelist; +int id; +int rank; +}; + +static DEFINE_MUTEX (memory_tier_lock); +static LIST_HEAD (memory_tiers); + +/* +* Keep it simple by having direct mapping between +* tier index and rank value. +*/ +static inline int get_rank_from_tier (unsigned int tier) +{ +switch (tier) { +case MEMORY_TIER_HBM_GPU : +return MEMORY_RANK_HBM_GPU ; +case MEMORY_TIER_DRAM : +return MEMORY_RANK_DRAM ; +case MEMORY_TIER_PMEM : +return MEMORY_RANK_PMEM ; + } +return -1 ; +} + +static void insert_memory_tier (struct memory_tier *memtier)Quoted 9 lines
mm/memory-tiers.c line 40
Quoted 6 lines
mm/memory-tiers.c new file +6
-0+{ +struct list_head *ent; +struct memory_tier *tmp_memtier; + +list_for_each (ent, &memory_tiers) { + tmp_memtier = list_entry(ent, struct memory_tier, list);
list_for_each_entry()?
Quoted 2 lines
ent variable is used below. Hence I won't be able to use
list_for_each_entry.
ent == &tmp_memtier->list ?
Quoted 14 lines
Quoted 13 lines
mm/memory-tiers.c line 48
Quoted 9 lines
mm/memory-tiers.c new file +8
-0+if (tmp_memtier->rank < memtier->rank) { +list_add_tail (&memtier->list, ent); +return ; + } + } +list_add_tail (&memtier->list, &memory_tiers); +} +IMHO, the locking requirements are needed here as comments to avoid confusing.
Quoted 5 lines
All those functions are called with
memory_tier_lock_held. Infact all list operations requires that lock held. What details do you suggest we document? I can add extra comment to the mutex itself? Adding locking details to all the functions will be duplicating the same details at multiple places?
memory_tier_lock isn't held to call register_memory_tier() in this patch. That will cause confusion.
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
mm/memory-tiers.c new file +34
-0+static struct memory_tier *register_memory_tier (unsigned int tier, +unsigned int rank) +{ +struct memory_tier *memtier; + +if (tier >=MAX_MEMORY_TIERS ) +return ERR_PTR (-EINVAL ); + + memtier =kzalloc (sizeof (struct memory_tier ),GFP_KERNEL ); +if (!memtier) +return ERR_PTR (-ENOMEM ); + + memtier->id = tier; + memtier->rank = rank; + +insert_memory_tier (memtier); + +return memtier; +} + +static int __init memory_tier_init (void ) +{ +struct memory_tier *memtier; + +/* +* Register only default memory tier to hide all empty +* memory tier from sysfs. +*/ + memtier =register_memory_tier (DEFAULT_MEMORY_TIER , +get_rank_from_tier (DEFAULT_MEMORY_TIER )); + +if (IS_ERR (memtier)) +panic ("%s() failed to register memory tier: %ld\n" , + __func__,PTR_ERR (memtier));
Quoted 8 lines
Quoted 8 lines
mm/memory-tiers.c line 89
Quoted 7 lines
mm/memory-tiers.c new file +7
-0+ +/* CPU only nodes are not part of memory tiers. */ + memtier->nodelist = node_states[N_MEMORY ]; + +return 0 ; +} +subsys_initcall (memory_tier_init);
Best Regards,
Huang, Ying
Johannes Weiner
maintainerRe: [PATCH v6 01/13] mm/demotion: Add support for explicit memory tiers
Quoted 10 lines · Ying Huang
Quoted 9 lines · Aneesh Kumar K V
Quoted 7 lines · Ying Huang
mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 619
Quoted 3 lines · Aneesh Kumar K.V
mm/Kconfig +3
-0+config TIERED_MEMORY + def_bool NUMA +As Yang pointed out, why not just use
CONFIG_NUMA? I suspect the added value ofCONIFIG_TIRED_MEMORY.
Quoted 6 lines
Quoted 3 lines
I decided to use
TIERED_MEMORYto bring more clarity. It should be same now that we have movedCONFIG_MIGRATIONdependencies to runtime. IMHO havingCONFIG_TIERED_MEMORYis better than usingCONFIG_NUMA.I don't think
CONFIG_TIERED_MEMORYbring no much value. It's better to useCONFIG_NUMAdirectly. But this is just my opinion.
I agree. As long as it's always built with CONFIG_NUMA, it's simply NUMA code. Easy enough to modularize it later if somebody really wants this to be configurable separately.
Re: [PATCH v6 01/13] mm/demotion: Add support for explicit memory tiers
Quoted 15 lines · Johannes Weiner
Quoted 14 lines · Ying Huang
Quoted 12 lines · Aneesh Kumar K V
Quoted 7 lines · Ying Huang
mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 619
Quoted 3 lines · Aneesh Kumar K.V
mm/Kconfig +3
-0+config TIERED_MEMORY + def_bool NUMA +As Yang pointed out, why not just use
CONFIG_NUMA? I suspect the added value ofCONIFIG_TIRED_MEMORY.I decided to use
TIERED_MEMORYto bring more clarity. It should be same now that we have movedCONFIG_MIGRATIONdependencies to runtime. IMHO havingCONFIG_TIERED_MEMORYis better than usingCONFIG_NUMA.
Quoted 6 lines
Quoted 2 lines
I don't think
CONFIG_TIERED_MEMORYbring no much value. It's better to useCONFIG_NUMAdirectly. But this is just my opinion.I agree. As long as it's always built with
CONFIG_NUMA, it's simply NUMA code. Easy enough to modularize it later if somebody really wants this to be configurable separately.
I was comparing,
#ifdef CONFIG_TIERED_MEMORY
struct memory_tier {
vs
#ifdef CONFIG_NUMA
struct memory_tier {
I will switch to CONFIG_NUMA in the next update since you are not finding it beneficial.
-aneesh
Re: [PATCH v6 01/13] mm/demotion: Add support for explicit memory tiers
In reply to Ying HuangYing Huang <[email protected]> writes:
....
Quoted 9 lines
Quoted 6 lines
All those functions are called with
memory_tier_lock_held. Infact all list operations requires that lock held. What details do you suggest we document? I can add extra comment to the mutex itself? Adding locking details to all the functions will be duplicating the same details at multiple places?
memory_tier_lockisn't held to callregister_memory_tier()in this patch. That will cause confusion.
will this help to explain this better
modified mm/memory-tiers.c
@@ -151,6 +151,11 @@ static void insert_memory_tier(struct memory_tier *memtier)
struct list_head *ent;
struct memory_tier *tmp_memtier;
+ if (IS_ENABLED(CONFIG_DEBUG_VM) && !mutex_is_locked(&memory_tier_lock)) {
+ WARN_ON_ONCE(1);
+ return;
+ }
+
list_for_each(ent, &memory_tiers) {
tmp_memtier = list_entry(ent, struct memory_tier, list); if (tmp_memtier->rank < memtier->rank) {
@@ -811,8 +816,12 @@ static int __init memory_tier_init(void)
/*
* Register only default memory tier to hide all empty
- * memory tier from sysfs.
+ * memory tier from sysfs. Since this is early during
+ * boot, we could avoid holding memtory_tier_lock. But
+ * keep it simple by holding locks. We can add lock
+ * held debug checks in other functions.
*/
+ mutex_lock(&memory_tier_lock);
memtier = register_memory_tier(DEFAULT_MEMORY_TIER,
get_rank_from_tier(DEFAULT_MEMORY_TIER));
@@ -828,6 +837,7 @@ static int __init memory_tier_init(void)
NODE_DATA(node)->memtier = memtier; node_set(node, memtier->nodelist);
}
+ mutex_unlock(&memory_tier_lock);
migrate_on_reclaim_init();
return 0;
-aneesh
Re: [PATCH v6 01/13] mm/demotion: Add support for explicit memory tiers
Review static void insert_memory_tier(struct memory_tier *memtier) · line 157
Quoted 13 lines
Quoted 2 lines
memory_tier_lockisn't held to callregister_memory_tier()in this patch. That will cause confusion.will this help to explain this better
modifiedmm/memory-tiers.c(quoted diff) +4
-0@@ -151,6 +151,11 @@ static void insert_memory_tier(struct memory_tier *memtier) struct list_head *ent; struct memory_tier *tmp_memtier; + if (IS_ENABLED(CONFIG_DEBUG_VM) && !mutex_is_locked(&memory_tier_lock)) { + WARN_ON_ONCE(1); + return; + }
Why not just use lockdep here instead?
PATCH v6 2/13 mm/demotion: Move memory demotion related code
In reply to Aneesh Kumar K.VThis move memory demotion related code to mm/memory-tiers.c. No functional change in this patch.
--- include/linux/memory-tiers.h | 6 ++++ include/linux/migrate.h | 2 -- mm/memory-tiers.c | 61 ++++++++++++++++++++++++++++++++++++ mm/migrate.c | 60 +---------------------------------- mm/vmscan.c | 1 + 5 files changed, 69 insertions(+), 61 deletions(-)
include/linux/memory-tiers.h +6 -0
@@ -2,6 +2,8 @@#ifndef _LINUX_MEMORY_TIERS_H#define _LINUX_MEMORY_TIERS_H +#include <linux/types.h> +#ifdef CONFIG_TIERED_MEMORY #define MEMORY_TIER_HBM_GPU 0
@@ -15,6 +17,10 @@#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM#define MAX_MEMORY_TIERS 3 +extern bool numa_demotion_enabled; +#else +#define numa_demotion_enabled false +#endif /* CONFIG_TIERED_MEMORY */ #endif
include/linux/migrate.h +0 -2
@@ -78,7 +78,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#if defined(CONFIG_MIGRATION ) && defined(CONFIG_NUMA )extern void set_migration_target_nodes (void );extern void migrate_on_reclaim_init (void ); -extern bool numa_demotion_enabled;extern int next_demotion_node (int node);#else static inline void set_migration_target_nodes (void ) {}
@@ -87,7 +86,6 @@ static inline int next_demotion_node(int node) {return NUMA_NO_NODE ; } -#define numa_demotion_enabled false#endif #ifdef CONFIG_COMPACTION
mm/memory-tiers.c +61 -0
69 lines not on this page · raw message
mm/migrate.c +1 -59
65 lines not on this page · raw message
mm/vmscan.c +1 -0
@@ -50,6 +50,7 @@#include <linux/printk.h> #include <linux/dax.h> #include <linux/psi.h> +#include <linux/memory-tiers.h> #include <asm/tlbflush.h> #include <asm/div64.h>
Signature · 1 line
2.36.1
PATCH v6 3/13 mm/demotion: Return error on write to numa_demotion sysfs
In reply to Aneesh Kumar K.VWith CONFIG_MIGRATION disabled return EINVAL on write.
--- mm/memory-tiers.c | 3 +++ 1 file changed, 3 insertions(+)
mm/memory-tiers.c +3 -0
@@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj, {ssize_t ret; +if (!IS_ENABLED (CONFIG_MIGRATION )) +return -EINVAL ; + ret =kstrtobool (buf, &numa_demotion_enabled);if (ret)return ret;
Signature · 1 line
2.36.1
Re: [PATCH v6 03/13] mm/demotion: Return error on write to numa_demotion sysfs
Quoted 16 lines · Aneesh Kumar K.V
With
CONFIG_MIGRATIONdisabled return EINVAL on write.Signed-off-by Aneesh Kumar K.V <[email protected]>---
mm/memory-tiers.c | 3 +++1 file changed, 3 insertions(+)mm/memory-tiers.c +0
-0@@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj, { ssize_t ret;
mm/memory-tiers.c static ssize_t numa_demotion_enabled_store(struct kobject *kobj, · line 110
Quoted 3 lines
mm/memory-tiers.c +3
-0+if (!IS_ENABLED (CONFIG_MIGRATION )) +return -EINVAL ; +
How about enclose numa_demotion_enabled_xxx related code with CONFIG_MIGRATION?
Quoted 3 lines
ret =
kstrtobool(buf, &numa_demotion_enabled); if (ret)
return ret;
Best Regards,
Huang, Ying
Re: [PATCH v6 03/13] mm/demotion: Return error on write to numa_demotion sysfs
Quoted 17 lines · Ying Huang
Quoted 16 lines · Aneesh Kumar K.V
With
CONFIG_MIGRATIONdisabled return EINVAL on write.Signed-off-by Aneesh Kumar K.V <[email protected]>--- mm/memory-tiers.c | 3 +++ 1 file changed, 3 insertions(+)mm/memory-tiers.c +0
-0@@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj, { ssize_t ret;
Quoted 6 lines
mm/memory-tiers.c static ssize_t numa_demotion_enabled_store(struct kobject *kobj, · line 110
Quoted 3 lines
mm/memory-tiers.c +3
-0+if (!IS_ENABLED (CONFIG_MIGRATION )) +return -EINVAL ; +How about enclose
numa_demotion_enabled_xxxrelated code withCONFIG_MIGRATION?
IIUC there is a desire to use IS_ENABLED() in the kernel instead of #ifdef since that helps in more compile time checks. Because there are no dead codes during compile now with IS_ENABLED().
W.r.t leaving the sysfs file visible even when CONFIG_MIGRATION is disabled, I was thinking it gives better visibility into numa_demotion status. I could switch to hide numa_demotion file if that is desirable.
Quoted 4 lines
Quoted 3 lines
mm/memory-tiers.c +0
-0ret = kstrtobool(buf, &numa_demotion_enabled); if (ret)return ret;
-aneesh
Re: [PATCH v6 03/13] mm/demotion: Return error on write to numa_demotion sysfs
Quoted 26 lines · Aneesh Kumar K V
Quoted 18 lines · Ying Huang
Quoted 17 lines · Aneesh Kumar K.V
With
CONFIG_MIGRATIONdisabled return EINVAL on write.Signed-off-by Aneesh Kumar K.V <[email protected]>---
mm/memory-tiers.c | 3 +++1 file changed, 3 insertions(+)mm/memory-tiers.c +0
-0@@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj, { ssize_t ret;Quoted 6 lines
mm/memory-tiers.c static ssize_t numa_demotion_enabled_store(struct kobject *kobj, · line 110
Quoted 3 lines
mm/memory-tiers.c +3
-0+if (!IS_ENABLED (CONFIG_MIGRATION )) +return -EINVAL ; +How about enclose
numa_demotion_enabled_xxxrelated code withCONFIG_MIGRATION?
Quoted 3 lines
IIUC there is a desire to use
IS_ENABLED()in the kernel instead of #ifdef since that helps in more compile time checks. Because there are no dead codes during compile now withIS_ENABLED().
IS_ENABLED() is used to reduce usage of "#ifdef" in ".c" file, especially inside a function. We have good build test coverage with 0Day now.
To avoid code size inflate, it's better to use #ifdef CONFIG_MIGRATION.
Quoted 8 lines
W.r.t leaving the sysfs file visible even when
CONFIG_MIGRATIONis disabled, I was thinking it gives better visibility intonuma_demotionstatus. I could switch to hidenuma_demotionfile if that is desirable.Quoted 4 lines
Quoted 3 lines
ret =
kstrtobool(buf, &numa_demotion_enabled); if (ret)
return ret;
Best Regards,
Huang, Ying
Re: [PATCH v6 03/13] mm/demotion: Return error on write to numa_demotion sysfs
Quoted 31 lines · Ying Huang
Quoted 26 lines · Aneesh Kumar K V
Quoted 18 lines · Ying Huang
Quoted 17 lines · Aneesh Kumar K.V
With
CONFIG_MIGRATIONdisabled return EINVAL on write.Signed-off-by Aneesh Kumar K.V <[email protected]>--- mm/memory-tiers.c | 3 +++ 1 file changed, 3 insertions(+)mm/memory-tiers.c +0
-0@@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj, { ssize_t ret;Quoted 6 lines
mm/memory-tiers.c static ssize_t numa_demotion_enabled_store(struct kobject *kobj, · line 110
Quoted 3 lines
mm/memory-tiers.c +3
-0+if (!IS_ENABLED (CONFIG_MIGRATION )) +return -EINVAL ; +How about enclose
numa_demotion_enabled_xxxrelated code withCONFIG_MIGRATION?Quoted 3 lines
IIUC there is a desire to use
IS_ENABLED()in the kernel instead of #ifdef since that helps in more compile time checks. Because there are no dead codes during compile now withIS_ENABLED().
Quoted 6 lines
IS_ENABLED()is used to reduce usage of "#ifdef" in ".c" file, especially inside a function. We have good build test coverage with 0Day now.To avoid code size inflate, it's better to use #ifdef
CONFIG_MIGRATION.
For a diff like below I am finding IS_ENABLED better.
size memory-tiers.o.isenabled memory-tiers.o
text data bss dec hex filename
4776 989 5 5770 168a memory-tiers.o.isenabled 5257 990 5 6252 186c memory-tiers.o
modified mm/memory-tiers.c
@@ -710,12 +710,11 @@ static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
static void __init migrate_on_reclaim_init(void)
{
-
- if (IS_ENABLED(CONFIG_MIGRATION)) {
+#ifdef CONFIG_MIGRATION
node_demotion = kcalloc(MAX_NUMNODES, sizeof(struct demotion_nodes),
GFP_KERNEL);
WARN_ON(!node_demotion);
- }
+#endif
hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
}
@@ -844,14 +843,19 @@ static ssize_t numa_demotion_enabled_show(struct kobject *kobj,
numa_demotion_enabled ? "true" : "false");
}
+#ifdef CONFIG_MIGRATION
static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
- ssize_t ret;
-
- if (!IS_ENABLED(CONFIG_MIGRATION))
- return -EINVAL;
+ return -EINVAL;
+}
+#else
+static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ ssize_t ret;
ret = kstrtobool(buf, &numa_demotion_enabled);
if (ret)
@@ -859,6 +863,7 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
return count;
}
+#endif
static struct kobj_attribute numa_demotion_enabled_attr =
__ATTR(demotion_enabled, 0644, numa_demotion_enabled_show,
I also find that #ifdef config not easier to the eyes. If there is a large code that we can end up #ifdef out, then it might be worth it. IIUC, we might want to keep the establish_migration target to find top_tier rank and lower_tier mask. Once we do that only thing that we could comment out is the node_demotion sysfs creation and I was considering to keep that even if migration is disabled with a write to the file returning EINVAL. I could switch that if you strongly feel that we should hide node_demotion sysfs file.
-aneesh
Re: [PATCH v6 03/13] mm/demotion: Return error on write to numa_demotion sysfs
Quoted 100 lines · Aneesh Kumar K V
Quoted 32 lines · Ying Huang
Quoted 27 lines · Aneesh Kumar K V
Quoted 19 lines · Ying Huang
Quoted 18 lines · Aneesh Kumar K.V
With
CONFIG_MIGRATIONdisabled return EINVAL on write.Signed-off-by Aneesh Kumar K.V <[email protected]>---
mm/memory-tiers.c | 3 +++1 file changed, 3 insertions(+)mm/memory-tiers.c +0
-0@@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj, { ssize_t ret;Quoted 6 lines
mm/memory-tiers.c static ssize_t numa_demotion_enabled_store(struct kobject *kobj, · line 110
Quoted 3 lines
mm/memory-tiers.c +3
-0+if (!IS_ENABLED (CONFIG_MIGRATION )) +return -EINVAL ; +How about enclose
numa_demotion_enabled_xxxrelated code withCONFIG_MIGRATION?Quoted 3 lines
IIUC there is a desire to use
IS_ENABLED()in the kernel instead of #ifdef since that helps in more compile time checks. Because there are no dead codes during compile now withIS_ENABLED().Quoted 6 lines
IS_ENABLED()is used to reduce usage of "#ifdef" in ".c" file, especially inside a function. We have good build test coverage with 0Day now.To avoid code size inflate, it's better to use #ifdef
CONFIG_MIGRATION.For a diff like below I am finding
IS_ENABLEDbetter.size memory-tiers.o.isenabled memory-tiers.o
text data bss dec hex filename4776 989 5 5770 168a memory-tiers.o.isenabled 5257 990 5 6252 186c memory-tiers.omodified
mm/memory-tiers.c(quoted diff) +11
-7@@ -710,12 +710,11 @@ static int __meminit migrate_on_reclaim_callback(struct notifier_block *self, static void __init migrate_on_reclaim_init(void) { - - if (IS_ENABLED(CONFIG_MIGRATION)) { +#ifdef CONFIG_MIGRATION node_demotion = kcalloc(MAX_NUMNODES, sizeof(struct demotion_nodes), GFP_KERNEL); WARN_ON(!node_demotion); - } +#endif hotplug_memory_notifier(migrate_on_reclaim_callback, 100); }@@ -844,14 +843,19 @@ static ssize_t numa_demotion_enabled_show(struct kobject *kobj, numa_demotion_enabled ? "true" : "false"); } +#ifdef CONFIG_MIGRATION static ssize_t numa_demotion_enabled_store(struct kobject *kobj, struct kobj_attribute *attr, const char *buf, size_t count) { - ssize_t ret; - - if (!IS_ENABLED(CONFIG_MIGRATION)) - return -EINVAL; + return -EINVAL; +} +#else +static ssize_t numa_demotion_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + ssize_t ret; ret = kstrtobool(buf, &numa_demotion_enabled);if (ret)
(quoted diff) +1
-0@@ -859,6 +863,7 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj, return count; } +#endifstatic struct
kobj_attributenuma_demotion_enabled_attr= __ATTR(demotion_enabled, 0644,numa_demotion_enabled_show,
Quoted 8 lines
I also find that #ifdef config not easier to the eyes. If there is a large code that we can end up #ifdef out, then it might be worth it. IIUC, we might want to keep the
establish_migrationtarget to findtop_tierrank andlower_tiermask. Once we do that only thing that we could comment out is thenode_demotionsysfs creation and I was considering to keep that even if migration is disabled with a write to the file returning EINVAL. I could switch that if you strongly feel that we should hidenode_demotionsysfs file.
Per my understanding, we can enclose most code about demoting/promoting inside CONFIG_MIGRATION, including numa/demotion_enabled sysfs interface. In this way, the code size can be reduced.
Best Regards,
Huang, Ying
PATCH v6 4/13 mm/demotion/dax/kmem: Set node's memory tier to MEMORY_TIER_PMEM
In reply to Aneesh Kumar K.VBy default, all nodes are assigned to DEFAULT_MEMORY_TIER which is the memory tier designated for nodes with DRAM
Set dax kmem device node's tier to MEMORY_TIER_PMEM. MEMORY_TIER_PMEM is assigned a default rank value of 100 and appears below DEFAULT_MEMORY_TIER in demotion order.
--- drivers/dax/kmem.c | 4 ++ include/linux/memory-tiers.h | 1 + mm/memory-tiers.c | 78 ++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+)
drivers/dax/kmem.c +4 -0
@@ -11,6 +11,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mman.h> +#include <linux/memory-tiers.h> #include "dax-private.h" #include "bus.h"
@@ -147,6 +148,9 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)dev_set_drvdata (dev, data); +#ifdef CONFIG_TIERED_MEMORY +node_create_and_set_memory_tier (numa_node,MEMORY_TIER_PMEM ); +#endif return 0 ; err_request_mem:
include/linux/memory-tiers.h +1 -0
@@ -18,6 +18,7 @@#define MAX_MEMORY_TIERS 3extern bool numa_demotion_enabled; +int node_create_and_set_memory_tier (int node,int tier);#else #define numa_demotion_enabled false
mm/memory-tiers.c +78 -0
84 lines not on this page · raw message
Signature · 1 line
2.36.1
Re: [PATCH v6 04/13] mm/demotion/dax/kmem: Set node's memory tier to MEMORY_TIER_PMEM
Quoted 65 lines · Aneesh Kumar K.V
By default, all nodes are assigned to
DEFAULT_MEMORY_TIERwhich is the memory tier designated for nodes with DRAMSet dax kmem device node's tier to
MEMORY_TIER_PMEM.MEMORY_TIER_PMEMis assigned a default rank value of 100 and appears belowDEFAULT_MEMORY_TIERin demotion order.Signed-off-by Jagdish Gediya <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>---
drivers/dax/kmem.c | 4 ++ include/linux/memory-tiers.h | 1 + mm/memory-tiers.c | 78 ++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+)drivers/dax/kmem.c +1
-0@@ -11,6 +11,7 @@ #include <linux/fs.h> #include <linux/mm.h> #include <linux/mman.h> +#include <linux/memory-tiers.h> #include "dax-private.h" #include "bus.h"drivers/dax/kmem.c +0
-0@@ -147,6 +148,9 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
dev_set_drvdata(dev, data);drivers/dax/kmem.c +3
-0+#ifdef CONFIG_TIERED_MEMORY +node_create_and_set_memory_tier (numa_node,MEMORY_TIER_PMEM ); +#endif return 0;
err_request_mem:
include/linux/memory-tiers.h +0
-0@@ -18,6 +18,7 @@ #define MAX_MEMORY_TIERS 3extern bool
numa_demotion_enabled;
+intnode_create_and_set_memory_tier(int node, int tier); #else
#definenuma_demotion_enabledfalsemm/memory-tiers.c +0
-0@@ -67,6 +67,84 @@ static struct memory_tier *register_memory_tier(unsigned int tier, return memtier; }mm/memory-tiers.c +1
-0+static struct memory_tier *__node_get_memory_tier (int node)
mm/memory-tiers.c static struct memory_tier *register_memory_tier(unsigned int tier, · line 80
Quoted 10 lines
mm/memory-tiers.c +10
-0+{ +struct memory_tier *memtier; + +list_for_each_entry (memtier, &memory_tiers, list) { +if (node_isset (node, memtier->nodelist)) +return memtier; + } +return NULL ; +} +
I suggest to add NODE_DATA(nid)->mem_tier before this patch. That is, part of [9/13]. That will make code much simpler and easier to review.
And, in addition to dax_kmem, whenever a normal node is onlined, we need to add it to the default memory tier. I found this is done in [5/13]. IMHO, we should move that part before this patch.
Best Regards,
Huang, Ying
Quoted 70 lines
mm/memory-tiers.c +67
-0+static struct memory_tier *__get_memory_tier_from_id (int id) +{ +struct memory_tier *memtier; + +list_for_each_entry (memtier, &memory_tiers, list) { +if (memtier->id == id) +return memtier; + } +return NULL ; +} + +static int __node_create_and_set_memory_tier (int node,int tier) +{ +int ret =0 ; +struct memory_tier *memtier; + + memtier =__get_memory_tier_from_id (tier); +if (!memtier) { +int rank; + + rank =get_rank_from_tier (tier); +if (rank ==-1 ) { + ret = -EINVAL ; + goto out; + } + memtier =register_memory_tier (tier, rank); +if (!memtier) { + ret = -EINVAL ; + goto out; + } + } +node_set (node, memtier->nodelist); +out: +return ret; +} + +int node_create_and_set_memory_tier (int node,int tier) +{ +struct memory_tier *current_tier; +int ret =0 ; + +mutex_lock (&memory_tier_lock); + + current_tier =__node_get_memory_tier (node); +if (!current_tier) { + ret =__node_create_and_set_memory_tier (node, tier); + goto out; + } + +if (current_tier->id == tier) + goto out; + +node_clear (node, current_tier->nodelist); + + ret =__node_create_and_set_memory_tier (node, tier); +if (ret) { +/* reset it back to older tier */ +node_set (node, current_tier->nodelist); + goto out; + } +out: +mutex_unlock (&memory_tier_lock); + +return ret; +} +EXPORT_SYMBOL_GPL (node_create_and_set_memory_tier); +static int
__initmemory_tier_init(void)
{
structmemory_tier*memtier;
Re: [PATCH v6 04/13] mm/demotion/dax/kmem: Set node's memory tier to MEMORY_TIER_PMEM
Quoted 65 lines · Ying Huang
Quoted 64 lines · Aneesh Kumar K.V
By default, all nodes are assigned to
DEFAULT_MEMORY_TIERwhich is the memory tier designated for nodes with DRAMSet dax kmem device node's tier to
MEMORY_TIER_PMEM.MEMORY_TIER_PMEMis assigned a default rank value of 100 and appears belowDEFAULT_MEMORY_TIERin demotion order.Signed-off-by Jagdish Gediya <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>--- drivers/dax/kmem.c | 4 ++ include/linux/memory-tiers.h | 1 + mm/memory-tiers.c | 78 ++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+)drivers/dax/kmem.c +4
-0@@ -11,6 +11,7 @@ #include <linux/fs.h> #include <linux/mm.h> #include <linux/mman.h> +#include <linux/memory-tiers.h> #include "dax-private.h" #include "bus.h"@@ -147,6 +148,9 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) dev_set_drvdata(dev, data); +#ifdef CONFIG_TIERED_MEMORY +node_create_and_set_memory_tier (numa_node,MEMORY_TIER_PMEM ); +#endif return 0;err_request_mem:
include/linux/memory-tiers.h +1
-0@@ -18,6 +18,7 @@ #define MAX_MEMORY_TIERS 3 extern bool numa_demotion_enabled; +int node_create_and_set_memory_tier (int node,int tier); #else #define numa_demotion_enabled falsemm/memory-tiers.c +0
-0@@ -67,6 +67,84 @@ static struct memory_tier *register_memory_tier(unsigned int tier, return memtier; }
Quoted 16 lines
mm/memory-tiers.c static struct memory_tier *register_memory_tier(unsigned int tier, · line 80
Quoted 11 lines
mm/memory-tiers.c +11
-0+static struct memory_tier *__node_get_memory_tier (int node) +{ +struct memory_tier *memtier; + +list_for_each_entry (memtier, &memory_tiers, list) { +if (node_isset (node, memtier->nodelist)) +return memtier; + } +return NULL ; +} +I suggest to add NODE_DATA(nid)->mem_tier before this patch. That is, part of [9/13]. That will make code much simpler and easier to review.
I would still like to keep introducing that NODE_DATA()->memtier later, so that the everything before that can be reviewed with simpler locking. __node_get_memory_tier() get fixed in patch 9 where all the details of that lockless access is documented. This also enables one to look at the history later.
Quoted 4 lines
And, in addition to
dax_kmem, whenever a normal node is onlined, we need to add it to the default memory tier. I found this is done in [5/13]. IMHO, we should move that part before this patch.
Sure will move the hotplug callback earlier.
-aneesh
PATCH v6 5/13 mm/demotion: Build demotion targets based on explicit memory tiers
In reply to Aneesh Kumar K.VThis patch switch the demotion target building logic to use memory tiers instead of NUMA distance. All N_MEMORY NUMA nodes will be placed in the default tier 1 and additional memory tiers will be added by drivers like dax kmem.
This patch builds the demotion target for a NUMA node by looking at all memory tiers below the tier to which the NUMA node belongs. The closest node in the immediately following memory tier is used as a demotion target.
Since we are now only building demotion target for N_MEMORY NUMA nodes the CPU hotplug calls are removed in this patch.
The rank approach allows us to keep memory tier device IDs stable even if there is a need to change the tier ordering among different memory tiers. e.g. DRAM nodes with CPUs will always be on memtier1, no matter how many tiers are higher or lower than these nodes. A new memory tier can be inserted into the tier hierarchy for a new set of nodes without affecting the node assignment of any existing memtier, provided that there is enough gap in the rank values for the new memtier.
The absolute value of "rank" of a memtier doesn't necessarily carry any meaning. Its value relative to other memtiers decides the level of this memtier in the tier hierarchy.
For now, This patch supports hardcoded rank values which are 300, 200, & 100 for memory tiers 0,1 & 2 respectively.
Below is the sysfs interface to read the rank values of memory tier, /sys/devices/system/memtier/memtierN/rank
This interface is read only for now. Write support can be added when there is a need of flexibility of more number of memory tiers(> 3) with flexibile ordering requirement among them.
--- include/linux/memory-tiers.h | 5 + include/linux/migrate.h | 13 -- mm/memory-tiers.c | 291 ++++++++++++++++++++++++++ mm/migrate.c | 394 ----------------------------------- mm/vmstat.c | 4 - 5 files changed, 296 insertions(+), 411 deletions(-)
include/linux/memory-tiers.h +5 -0
@@ -19,8 +19,13 @@extern bool numa_demotion_enabled;int node_create_and_set_memory_tier (int node,int tier); +int next_demotion_node (int node);#else #define numa_demotion_enabled false +static inline int next_demotion_node (int node) +{ +return NUMA_NO_NODE ; +}#endif /* CONFIG_TIERED_MEMORY */
include/linux/migrate.h +0 -13
@@ -75,19 +75,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ -#if defined(CONFIG_MIGRATION ) && defined(CONFIG_NUMA ) -extern void set_migration_target_nodes (void ); -extern void migrate_on_reclaim_init (void ); -extern int next_demotion_node (int node); -#else -static inline void set_migration_target_nodes (void ) {} -static inline void migrate_on_reclaim_init (void ) {} -static inline int next_demotion_node (int node) -{ -return NUMA_NO_NODE ; -} -#endif -#ifdef CONFIG_COMPACTION extern int PageMovable (struct page *page);extern void __SetPageMovable (struct page *page,struct address_space *mapping);
mm/memory-tiers.c +291 -0
330 lines not on this page · raw message
mm/migrate.c +0 -394
398 lines not on this page · raw message
mm/vmstat.c +0 -4
@@ -28,7 +28,6 @@#include <linux/mm_inline.h> #include <linux/page_ext.h> #include <linux/page_owner.h> -#include <linux/migrate.h> #include "internal.h"
@@ -2060,7 +2059,6 @@ static int vmstat_cpu_online(unsigned int cpu)if (!node_state (cpu_to_node (cpu),N_CPU )) {node_set_state (cpu_to_node (cpu),N_CPU ); -set_migration_target_nodes (); }return 0 ;
@@ -2085,7 +2083,6 @@ static int vmstat_cpu_dead(unsigned int cpu)return 0 ;node_clear_state (node,N_CPU ); -set_migration_target_nodes ();return 0 ; }
@@ -2118,7 +2115,6 @@ void __init init_mm_internals(void)start_shepherd_timer ();#endif -migrate_on_reclaim_init ();#ifdef CONFIG_PROC_FS proc_create_seq ("buddyinfo" ,0444 ,NULL , &fragmentation_op);proc_create_seq ("pagetypeinfo" ,0400 ,NULL , &pagetypeinfo_op);
Signature · 1 line
2.36.1
PATCH v6 6/13 mm/demotion: Expose memory tier details via sysfs
In reply to Aneesh Kumar K.VThis patch adds /sys/devices/system/memtier/ where all memory tier related details can be found. All created memory tiers will be listed there as /sys/devices/system/memtier/memtierN/
The nodes which are part of a specific memory tier can be listed via /sys/devices/system/memtier/memtierN/nodelist
The rank value of a memory tier can be listed via
via /sys/devices/system/memtier/memtierN/rank
/sys/devices/system/memtier/max_tier shows the maximum number of memory tiers that can be created.
/sys/devices/system/memtier/default_tier shows the memory tier to which NUMA nodes get added by default if not assigned a specific memory tier.
--- mm/memory-tiers.c | 99 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 95 insertions(+), 4 deletions(-)
mm/memory-tiers.c +95 -4
@@ -11,8 +11,8 @@struct memory_tier {struct list_head list; +struct device dev;nodemask_t nodelist; -int id;int rank; };
@@ -20,6 +20,7 @@ struct demotion_nodes {nodemask_t preferred; }; +#define to_memory_tier (device) container_of(device, struct memory_tier, dev)static void establish_migration_targets (void );static DEFINE_MUTEX (memory_tier_lock);static LIST_HEAD (memory_tiers);
@@ -86,6 +87,52 @@ static LIST_HEAD(memory_tiers);*/ static struct demotion_nodes *node_demotion__read_mostly ; +static struct bus_type memory_tier_subsys = { + .name ="memtier" , + .dev_name ="memtier" , +}; + +static ssize_t nodelist_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct memory_tier *memtier =to_memory_tier (dev); + +return sysfs_emit (buf,"%*pbl\n" , +nodemask_pr_args (&memtier->nodelist)); +} +static DEVICE_ATTR_RO (nodelist); + +static ssize_t rank_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct memory_tier *memtier =to_memory_tier (dev); + +return sysfs_emit (buf,"%d\n" , memtier->rank); +} +static DEVICE_ATTR_RO (rank); + +static struct attribute *memory_tier_dev_attrs[] = { + &dev_attr_nodelist.attr, + &dev_attr_rank.attr, +NULL +}; + +static const struct attribute_group memory_tier_dev_group = { + .attrs = memory_tier_dev_attrs, +}; + +static const struct attribute_group *memory_tier_dev_groups[] = { + &memory_tier_dev_group, +NULL +}; + +static void memory_tier_device_release (struct device *dev) +{ +struct memory_tier *tier =to_memory_tier (dev); + +kfree (tier); +} +/* * Keep it simple by having direct mapping between * tier index and rank value.
@@ -121,6 +168,7 @@ static void insert_memory_tier(struct memory_tier *memtier)static struct memory_tier *register_memory_tier (unsigned int tier,unsigned int rank) { +int error;struct memory_tier *memtier;if (tier >=MAX_MEMORY_TIERS )
@@ -130,11 +178,20 @@ static struct memory_tier *register_memory_tier(unsigned int tier,if (!memtier)return ERR_PTR (-ENOMEM ); - memtier->id = tier; + memtier->dev.id = tier; memtier->rank = rank; + memtier->dev.bus = &memory_tier_subsys; + memtier->dev.release = memory_tier_device_release; + memtier->dev.groups = memory_tier_dev_groups;insert_memory_tier (memtier); + error =device_register (&memtier->dev); +if (error) { +list_del (&memtier->list); +put_device (&memtier->dev); +return ERR_PTR (error); + }return memtier; }
@@ -154,7 +211,7 @@ static struct memory_tier *__get_memory_tier_from_id(int id)struct memory_tier *memtier;list_for_each_entry (memtier, &memory_tiers, list) { -if (memtier->id == id) +if (memtier->dev.id == id)return memtier; }return NULL ;
@@ -199,7 +256,7 @@ int node_create_and_set_memory_tier(int node, int tier) goto out; } -if (current_tier->id == tier) +if (current_tier->dev.id == tier) goto out;node_clear (node, current_tier->nodelist);
@@ -435,10 +492,44 @@ static void __init migrate_on_reclaim_init(void)hotplug_memory_notifier (migrate_on_reclaim_callback,100 ); } +static ssize_t +max_tier_show (struct device *dev,struct device_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"%d\n" ,MAX_MEMORY_TIERS ); +} +static DEVICE_ATTR_RO (max_tier); + +static ssize_t +default_tier_show (struct device *dev,struct device_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"memtier%d\n" ,DEFAULT_MEMORY_TIER ); +} +static DEVICE_ATTR_RO (default_tier); + +static struct attribute *memory_tier_attrs[] = { + &dev_attr_max_tier.attr, + &dev_attr_default_tier.attr, +NULL +}; + +static const struct attribute_group memory_tier_attr_group = { + .attrs = memory_tier_attrs, +}; + +static const struct attribute_group *memory_tier_attr_groups[] = { + &memory_tier_attr_group, +NULL , +}; +static int __init memory_tier_init (void ) { +int ret;struct memory_tier *memtier; + ret =subsys_system_register (&memory_tier_subsys, memory_tier_attr_groups); +if (ret) +pr_err ("%s() failed to register subsystem: %d\n" , __func__, ret); +/* * Register only default memory tier to hide all empty * memory tier from sysfs.
Signature · 1 line
2.36.1
PATCH v6 7/13 mm/demotion: Add per node memory tier attribute to sysfs
In reply to Aneesh Kumar K.VAdd support to modify the memory tier for a NUMA node.
/sys/devices/system/node/nodeN/memtier
where N = node id
When read, It list the memory tier that the node belongs to.
When written, the kernel moves the node into the specified memory tier, the tier assignment of all other nodes are not affected.
If the memory tier does not exist an error is returned.
--- drivers/base/node.c | 39 ++++++++++++++++++++++++++++++++ include/linux/memory-tiers.h | 3 +++ mm/memory-tiers.c | 44 ++++++++++++++++++++++++++++++++++++ 3 files changed, 86 insertions(+)
drivers/base/node.c +39 -0
@@ -20,6 +20,7 @@#include <linux/pm_runtime.h> #include <linux/swap.h> #include <linux/slab.h> +#include <linux/memory-tiers.h> static struct bus_type node_subsys = { .name ="node" ,
@@ -560,11 +561,49 @@ static ssize_t node_read_distance(struct device *dev, } static DEVICE_ATTR(distance, 0444, node_read_distance, NULL); +#ifdef CONFIG_TIERED_MEMORY +static ssize_t memtier_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +int node = dev->id; +int tier_index =node_get_memory_tier_id (node); + +if (tier_index !=-1 ) +return sysfs_emit (buf,"%d\n" , tier_index); +return 0 ; +} + +static ssize_t memtier_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +unsigned long tier; +int node = dev->id; +int ret; + + ret =kstrtoul (buf,10 , &tier); +if (ret) +return ret; + + ret =node_reset_memory_tier (node, tier); +if (ret) +return ret; + +return count; +} + +static DEVICE_ATTR_RW (memtier); +#endif +static struct attribute *node_dev_attrs[] = { &dev_attr_meminfo.attr, &dev_attr_numastat.attr, &dev_attr_distance.attr, &dev_attr_vmstat.attr, +#ifdef CONFIG_TIERED_MEMORY + &dev_attr_memtier.attr, +#endifNULL };
include/linux/memory-tiers.h +3 -0
@@ -20,6 +20,9 @@extern bool numa_demotion_enabled;int node_create_and_set_memory_tier (int node,int tier);int next_demotion_node (int node); +int node_set_memory_tier (int node,int tier); +int node_get_memory_tier_id (int node); +int node_reset_memory_tier (int node,int tier);#else #define numa_demotion_enabled falsestatic inline int next_demotion_node (int node)
mm/memory-tiers.c +44 -0
50 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v6 8/13 mm/demotion: Add support for memory tier creation from userspace
In reply to Aneesh Kumar K.VThis patch adds support to create memory tiers with specific rank value from userspace. To avoid race while creating memory tiers /sys/devices/system/memtier/create_tier_from_rank file is provided. Writing to this file with a specific rank value creates a new memory tier with the specified rank value.
Memory tiers created from userspace gets destroyed when the memory tier nodelist becomes empty.
--- include/linux/memory-tiers.h | 3 +- mm/memory-tiers.c | 74 ++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+), 1 deletion(-)
include/linux/memory-tiers.h +2 -1
@@ -15,7 +15,8 @@#define MEMORY_RANK_PMEM 100#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM -#define MAX_MEMORY_TIERS 3 +#define MAX_STATIC_MEMORY_TIERS 3 +#define MAX_MEMORY_TIERS (MAX_STATIC_MEMORY_TIERS + 2)extern bool numa_demotion_enabled;int node_create_and_set_memory_tier (int node,int tier);
mm/memory-tiers.c +74 -0
116 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v6 9/13 mm/demotion: Add pg_data_t member to track node memory tier details
In reply to Aneesh Kumar K.VAlso update different helpes to use NODE_DATA()->memtier. Since node specific memtier can change based on the reassignment of NUMA node to a different memory tiers, accessing NODE_DATA()->memtier needs to under an rcu read lock of memory_tier_lock.
--- include/linux/memory-tiers.h | 14 +++++ include/linux/mmzone.h | 3 ++ mm/memory-tiers.c | 102 ++++++++++++++++++++++++++--------- 3 files changed, 94 insertions(+), 25 deletions(-)
include/linux/memory-tiers.h +14 -0
@@ -6,6 +6,9 @@#ifdef CONFIG_TIERED_MEMORY +#include <linux/device.h> +#include <linux/nodemask.h> +#define MEMORY_TIER_HBM_GPU 0#define MEMORY_TIER_DRAM 1#define MEMORY_TIER_PMEM 2
@@ -18,13 +21,24 @@#define MAX_STATIC_MEMORY_TIERS 3#define MAX_MEMORY_TIERS (MAX_STATIC_MEMORY_TIERS + 2) +struct memory_tier { +struct list_head list; +struct device dev; +nodemask_t nodelist; +int rank; +}; +extern bool numa_demotion_enabled;int node_create_and_set_memory_tier (int node,int tier);int next_demotion_node (int node);int node_set_memory_tier (int node,int tier);int node_get_memory_tier_id (int node);int node_reset_memory_tier (int node,int tier); +struct memory_tier *node_get_memory_tier (int node); +void node_put_memory_tier (struct memory_tier *memtier); +#else +#define numa_demotion_enabled falsestatic inline int next_demotion_node (int node) {
include/linux/mmzone.h +3 -0
@@ -928,6 +928,9 @@ typedef struct pglist_data {/* Per-node vmstats */ struct per_cpu_nodestat __percpu *per_cpu_nodestats;atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS ]; +#ifdef CONFIG_TIERED_MEMORY +struct memory_tier *memtier; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)
mm/memory-tiers.c +77 -25
176 lines not on this page · raw message
Signature · 1 line
2.36.1
Re: [PATCH v6 09/13] mm/demotion: Add pg_data_t member to track node memory tier details
Quoted 33 lines · Aneesh Kumar K.V
Also update different helpes to use NODE_DATA()->memtier. Since node specific memtier can change based on the reassignment of NUMA node to a different memory tiers, accessing NODE_DATA()->memtier needs to under an rcu read lock of
memory_tier_lock.Signed-off-by Aneesh Kumar K.V <[email protected]>---
include/linux/memory-tiers.h | 14 +++++ include/linux/mmzone.h | 3 ++ mm/memory-tiers.c | 102 ++++++++++++++++++++++++++--------- 3 files changed, 94 insertions(+), 25 deletions(-)include/linux/memory-tiers.h +0
-0@@ -6,6 +6,9 @@#ifdef
CONFIG_TIERED_MEMORYinclude/linux/memory-tiers.h +3
-0+#include <linux/device.h> +#include <linux/nodemask.h> +#define
MEMORY_TIER_HBM_GPU0
#defineMEMORY_TIER_DRAM1
#defineMEMORY_TIER_PMEM2include/linux/memory-tiers.h +0
-0@@ -18,13 +21,24 @@ #define MAX_STATIC_MEMORY_TIERS 3 #define MAX_MEMORY_TIERS (MAX_STATIC_MEMORY_TIERS + 2)
include/linux/memory-tiers.h line 30
Quoted 7 lines
include/linux/memory-tiers.h +7
-0+struct memory_tier { +struct list_head list; +struct device dev; +nodemask_t nodelist; +int rank; +}; +
I suggest to use two data structure,
struct memory_tier {
struct list_head list;
nodemask_t nodelist;
int rank;
};struct memory_tier_dev {
struct list_head list;
struct device dev;
struct memory_tier *tier;
};Then we can put struct memory_tier here and still hide struct memory_tier_dev in memory_tiers.c. In this way, we don't need to force all struct memory_tier users to compile the entire driver core headers. And we can separate the user space interface implementation from the other part of the kernel.
include/linux/memory-tiers.h line 38
Quoted 8 lines
extern bool
numa_demotion_enabled;
intnode_create_and_set_memory_tier(int node, int tier); intnext_demotion_node(int node);
intnode_set_memory_tier(int node, int tier); intnode_get_memory_tier_id(int node);
intnode_reset_memory_tier(int node, int tier);include/linux/memory-tiers.h +2
-0+struct memory_tier *node_get_memory_tier (int node); +void node_put_memory_tier (struct memory_tier *memtier);
I don't find caller of these 2 functions in series. Can we remove these functions?
Best Regards,
Huang, Ying
[snip]
PATCH v6 10/13 mm/demotion: Demote pages according to allocation fallback order
In reply to Aneesh Kumar K.VFrom: Jagdish Gediya <[email protected]>
Currently, a higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space). This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from any lower tier, whereas the demotion order doesn't allow that currently.
This patch adds support to get all the allowed demotion targets for a memory tier. demote_page_list() function is now modified to utilize this allowed node mask as the fallback allocation mask.
move allowed mask to memory tier
--- include/linux/memory-tiers.h | 9 ++++- mm/memory-tiers.c | 75 +++++++++++++++++++++++++++++++++--- mm/vmscan.c | 56 ++++++++++++++++++++------- 3 files changed, 120 insertions(+), 20 deletions(-)
include/linux/memory-tiers.h +8 -1
@@ -3,11 +3,12 @@#define _LINUX_MEMORY_TIERS_H#include <linux/types.h> +#include <linux/nodemask.h> +#include <linux/mmzone.h> #ifdef CONFIG_TIERED_MEMORY #include <linux/device.h> -#include <linux/nodemask.h> #define MEMORY_TIER_HBM_GPU 0#define MEMORY_TIER_DRAM 1
@@ -25,6 +26,7 @@ struct memory_tier {struct list_head list;struct device dev;nodemask_t nodelist; +nodemask_t lower_tier_mask;int rank; };
@@ -36,6 +38,7 @@ int node_get_memory_tier_id(int node);int node_reset_memory_tier (int node,int tier);struct memory_tier *node_get_memory_tier (int node);void node_put_memory_tier (struct memory_tier *memtier); +void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets);#else
@@ -45,6 +48,10 @@ static inline int next_demotion_node(int node)return NUMA_NO_NODE ; } +static inline void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets) +{ + *targets =NODE_MASK_NONE ; +}#endif /* CONFIG_TIERED_MEMORY */ #endif
mm/memory-tiers.c +70 -5
102 lines not on this page · raw message
mm/vmscan.c +42 -14
77 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v6 11/13 mm/demotion: Update node_is_toptier to work with memory tiers
In reply to Aneesh Kumar K.VWith memory tiers support we can have memory only NUMA nodes in the top tier from which we want to avoid promotion tracking NUMA faults. Update node_is_toptier to work with memory tiers. All NUMA nodes are by default top tier nodes. With lower memory tiers added we consider all memory tiers above a memory tier having CPU NUMA nodes as top memory tier
--- include/linux/memory-tiers.h | 6 +++++ include/linux/node.h | 5 ---- mm/huge_memory.c | 1 + mm/memory-tiers.c | 44 ++++++++++++++++++++++++++++++++++-- mm/migrate.c | 1 + mm/mprotect.c | 1 + 6 files changed, 51 insertions(+), 7 deletions(-)
include/linux/memory-tiers.h +6 -0
@@ -39,6 +39,7 @@ int node_reset_memory_tier(int node, int tier);struct memory_tier *node_get_memory_tier (int node);void node_put_memory_tier (struct memory_tier *memtier);void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets); +bool node_is_toptier (int node);#else
@@ -52,6 +53,11 @@ static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *target { *targets =NODE_MASK_NONE ; } + +static inline bool node_is_toptier (int node) +{ +return true; +}#endif /* CONFIG_TIERED_MEMORY */ #endif
include/linux/node.h +0 -5
@@ -185,9 +185,4 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,#define to_node (device) container_of(device, struct node, dev) -static inline bool node_is_toptier (int node) -{ -return node_state (node,N_CPU ); -} -#endif /* _LINUX_NODE_H_ */
mm/huge_memory.c +1 -0
@@ -35,6 +35,7 @@#include <linux/numa.h> #include <linux/page_owner.h> #include <linux/sched/sysctl.h> +#include <linux/memory-tiers.h> #include <asm/tlb.h> #include <asm/pgalloc.h>
mm/memory-tiers.c +42 -2
68 lines not on this page · raw message
mm/migrate.c +1 -0
@@ -50,6 +50,7 @@#include <linux/memory.h> #include <linux/random.h> #include <linux/sched/sysctl.h> +#include <linux/memory-tiers.h> #include <asm/tlbflush.h>
mm/mprotect.c +1 -0
@@ -31,6 +31,7 @@#include <linux/pgtable.h> #include <linux/sched/sysctl.h> #include <linux/userfaultfd_k.h> +#include <linux/memory-tiers.h> #include <asm/cacheflush.h> #include <asm/mmu_context.h> #include <asm/tlbflush.h>
Signature · 1 line
2.36.1
PATCH v6 12/13 mm/demotion: Add documentation for memory tiering
In reply to Aneesh Kumar K.VFrom: Jagdish Gediya <[email protected]>
All N_MEMORY nodes are divided into 3 memoty tiers with rank value MEMORY_RANK_HBM_GPU, MEMORY_RANK_DRAM and MEMORY_RANK_PMEM. By default, All nodes are assigned to memory tier with rank value MEMORY_RANK_DRAM.
Demotion path for all N_MEMORY nodes is prepared based on the rank value of memory tiers.
This patch adds documention for memory tiering introduction, its sysfs interfaces and how demotion is performed based on memory tiers.
--- Documentation/admin-guide/mm/index.rst | 1 + .../admin-guide/mm/memory-tiering.rst | 181 ++++++++++++++++++ 2 files changed, 182 insertions(+) create mode 100644 Documentation/admin-guide/mm/memory-tiering.rst
Documentation/admin-guide/mm/index.rst +1 -0
@@ -32,6 +32,7 @@ the Linux memory management. idle_page_tracking ksm memory-hotplug + memory-tiering nommu-mmap numa_memory_policy numaperf
Documentation/admin-guide/mm/memory-tiering.rst new file +181 -0
181 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v6 13/13 mm/demotion: Add sysfs ABI documentation
In reply to Aneesh Kumar K.VAdd sysfs ABI documentation.
--- .../ABI/testing/sysfs-kernel-mm-memory-tiers | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers new file +87 -0
@@ -0,0 +1,87 @@ +What: /sys/devices/system/memtier/ +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Interface for tiered memory + + This is the directory containing the information about memory tiers. + + Each memory tier has its own subdirectory. + + The order of memory tiers is determined by their rank values, not by + their memtier device names. A higher rank value means a higher tier. + +What: /sys/devices/system/memtier/default_tier +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Default memory tier + + The default memory tier to which memory would get added via hotplug + if the NUMA node is not part of any memory tier + +What: /sys/devices/system/memtier/max_tier +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Maximum number of memory tiers supported + + The max memory tier device ID we can create. Users can create memory + tiers in range [0 - max_tier) + +What: /sys/devices/system/memtier/create_tier_from_rank +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Interface to create memory tiers from userspace + + Writing to this file with a rank value results in creation of + a new memory tier with the specified rank value. This is used + by userspace to create new memory tiers. + +What: /sys/devices/system/memtier/memtierN/ +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Directory with details of a specific memory tier + + This is the directory containing the information about a particular + memory tier, memtierN, where N is the memtier device ID (e.g. 0, 1). + + The memtier device ID number itself is just an identifier and has no + special meaning, i.e. memtier device ID numbers do not determine the + order of memory tiers. + +What: /sys/devices/system/memtier/memtierN/rank +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Memory tier rank value + + + When read, list the "rank" value associated with memtierN. + + "Rank" is an opaque value. Its absolute value doesn't have any + special meaning. But the rank values of different memtiers can be + compared with each other to determine the memory tier order. + + For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and + their rank values are 100, 10, 50, then the memory tier order is: + memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier + and memtier1 is the lowest tier. + + The rank value of each memtier should be unique. + +What: /sys/devices/system/memtier/memtierN/nodelist +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Memory tier nodelist + + + When read, list the memory nodes in the specified tier. + +What: /sys/devices/system/node/nodeN/memtier +Date: June 2022 +Contact: Linux memory management mailing list <[email protected]> +Description: Memory tier details for node N + + When read, list the device ID of the memory tier that the node belongs + to. Its value is empty for a CPU-only NUMA node. + + When written, the kernel moves the node into the specified memory + tier if the move is allowed. The tier assignments of all other + nodes are not affected.
Signature · 1 line
2.36.1