mm/demotion: Memory tiers and demotion

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1,

where memtier0 is the highest tier
and memtier1 is the lowest tier.

mm/memory-tiers.c new file +3 -0

+	kfree(tier);
+}
+

mm/memory-tiers.c new file +8 -0

+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+	int error;
+	struct memory_tier *memtier;
+
+	if (tier >= MAX_MEMORY_TIERS)
+		return NULL;
+

mm/memory-tiers.c new file +4 -0

+	insert_memory_tier(memtier);
+
+	error = device_register(&memtier->dev);
+	if (error) {

Quoted 10 lines · Aneesh Kumar K.V

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

Quoted 3 lines

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1,

Why is memtier2 (rank 100) higher than memtier1 (rank 200)? Seems like the order should be memtier0 -> memtier1 -> memtier2?

                    (rank 300)  (rank 200)  (rank 100)

Quoted 2 lines

where memtier0 is the highest tier
and memtier1 is the lowest tier.

I think memtier2 is the lowest as it has the lowest rank value.

mm/memory-tiers.c line 63

Quoted 9 lines

The rank value of each memtier should be unique.

mm/memory-tiers.c new file +5 -0

+
+static void memory_tier_device_release(struct device *dev)
+{
+     struct memory_tier *tier = to_memory_tier(dev);
+

Do we need some ref counts on memory_tier?
If there is another device still using the same memtier, free below could cause problem.

Quoted 4 lines

mm/memory-tiers.c new file +3 -0

+     kfree(tier);
+}
+

...

Quoted 8 lines

mm/memory-tiers.c new file +8 -0

+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+     int error;
+     struct memory_tier *memtier;
+
+     if (tier >= MAX_MEMORY_TIERS)
+             return NULL;
+

mm/memory-tiers.c line 117

Quoted 10 lines

mm/memory-tiers.c new file +10 -0

+     memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+     if (!memtier)
+             return NULL;
+
+     memtier->dev.id = tier;
+     memtier->rank = get_rank_from_tier(tier);
+     memtier->dev.bus = &memory_tier_subsys;
+     memtier->dev.release = memory_tier_device_release;
+     memtier->dev.groups = memory_tier_dev_groups;
+

Should you take the mem_tier_lock before you insert to memtier-list?

Quoted 4 lines

mm/memory-tiers.c new file +4 -0

+     insert_memory_tier(memtier);
+
+     error = device_register(&memtier->dev);
+     if (error) {

mm/memory-tiers.c line 131

Quoted 10 lines

mm/memory-tiers.c new file +10 -0

+             list_del(&memtier->list);
+             put_device(&memtier->dev);
+             return NULL;
+     }
+     return memtier;
+}
+
+__maybe_unused // temporay to prevent warnings during bisects
+static void unregister_memory_tier(struct memory_tier *memtier)
+{

I think we should take mem_tier_lock before modifying memtier->list.

mm/memory-tiers.c line 135

Quoted 5 lines

mm/memory-tiers.c new file +4 -0

+     list_del(&memtier->list);
+     device_unregister(&memtier->dev);
+}
+

Thanks.

Tim

Quoted 6 lines · Aneesh Kumar K.V

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

Quoted 7 lines

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1,

Why is memtier2 (rank 100) higher than memtier1 (rank 200)? Seems like the order should be memtier0 -> memtier1 -> memtier2?

                     (rank 300)  (rank 200)  (rank 100)

Quoted 2 lines

where memtier0 is the highest tier
and memtier1 is the lowest tier.

I think memtier2 is the lowest as it has the lowest rank value.

mm/memory-tiers.c line 63

Quoted 9 lines

The rank value of each memtier should be unique.

mm/memory-tiers.c new file +5 -0

+
+static void memory_tier_device_release(struct device *dev)
+{
+	struct memory_tier *tier = to_memory_tier(dev);
+

Do we need some ref counts on memory_tier?
If there is another device still using the same memtier, free below could cause problem.

Quoted 4 lines

mm/memory-tiers.c new file +3 -0

+	kfree(tier);
+}
+

...

Quoted 13 lines

mm/memory-tiers.c new file +13 -0

+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+	int error;
+	struct memory_tier *memtier;
+
+	if (tier >= MAX_MEMORY_TIERS)
+		return NULL;
+
+	memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+	if (!memtier)
+		return NULL;
+
+	memtier->dev.id = tier;

mm/memory-tiers.c line 117

Quoted 5 lines

mm/memory-tiers.c new file +5 -0

+	memtier->rank = get_rank_from_tier(tier);
+	memtier->dev.bus = &memory_tier_subsys;
+	memtier->dev.release = memory_tier_device_release;
+	memtier->dev.groups = memory_tier_dev_groups;
+

Should you take the mem_tier_lock before you insert to memtier-list?

Quoted 5 lines

mm/memory-tiers.c new file +4 -0

+	insert_memory_tier(memtier);
+
+	error = device_register(&memtier->dev);
+	if (error) {

mm/memory-tiers.c line 131

Quoted 10 lines

mm/memory-tiers.c new file +10 -0

+		list_del(&memtier->list);
+		put_device(&memtier->dev);
+		return NULL;
+	}
+	return memtier;
+}
+
+__maybe_unused // temporay to prevent warnings during bisects
+static void unregister_memory_tier(struct memory_tier *memtier)
+{

I think we should take mem_tier_lock before modifying memtier->list.

mm/memory-tiers.c line 135

Quoted 5 lines

mm/memory-tiers.c new file +4 -0

+	list_del(&memtier->list);
+	device_unregister(&memtier->dev);
+}
+

Quoted 15 lines · Tim Chen

Quoted 6 lines · Aneesh Kumar K.V

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

Quoted 7 lines

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1,

Quoted 8 lines

Why is memtier2 (rank 100) higher than memtier1 (rank 200)? Seems like the order should be memtier0 -> memtier1 -> memtier2?
                     (rank 300) (rank 200) (rank 100)

Quoted 2 lines

where memtier0 is the highest tier
and memtier1 is the lowest tier.

I think memtier2 is the lowest as it has the lowest rank value.

typo error. Will fix that in the next update

Quoted 32 lines

mm/memory-tiers.c line 63

Quoted 9 lines

The rank value of each memtier should be unique.

mm/memory-tiers.c new file +5 -0

+
+static void memory_tier_device_release(struct device *dev)
+{
+	struct memory_tier *tier = to_memory_tier(dev);
+

Do we need some ref counts on memory_tier?
If there is another device still using the same memtier, free below could cause problem.

Quoted 4 lines

mm/memory-tiers.c new file +3 -0

+	kfree(tier);
+}
+

...

Quoted 13 lines

mm/memory-tiers.c new file +13 -0

+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+	int error;
+	struct memory_tier *memtier;
+
+	if (tier >= MAX_MEMORY_TIERS)
+		return NULL;
+
+	memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+	if (!memtier)
+		return NULL;
+
+	memtier->dev.id = tier;

Quoted 8 lines

mm/memory-tiers.c line 117

Quoted 5 lines

mm/memory-tiers.c new file +5 -0

+	memtier->rank = get_rank_from_tier(tier);
+	memtier->dev.bus = &memory_tier_subsys;
+	memtier->dev.release = memory_tier_device_release;
+	memtier->dev.groups = memory_tier_dev_groups;
+

Should you take the mem_tier_lock before you insert to memtier-list?

Both register_memory_tier and unregister_memory_tier get called with memory_tier_lock held.

Quoted 18 lines

Quoted 5 lines

mm/memory-tiers.c new file +4 -0

+	insert_memory_tier(memtier);
+
+	error = device_register(&memtier->dev);
+	if (error) {

mm/memory-tiers.c line 131

Quoted 10 lines

mm/memory-tiers.c new file +10 -0

+		list_del(&memtier->list);
+		put_device(&memtier->dev);
+		return NULL;
+	}
+	return memtier;
+}
+
+__maybe_unused // temporay to prevent warnings during bisects
+static void unregister_memory_tier(struct memory_tier *memtier)
+{

I think we should take mem_tier_lock before modifying memtier->list.

unregister_memory_tier get called with memory_tier_lock held.

Quoted 5 lines

mm/memory-tiers.c line 135

Quoted 5 lines

mm/memory-tiers.c new file +4 -0

+	list_del(&memtier->list);
+	device_unregister(&memtier->dev);
+}
+

-aneesh

mm/memory-tiers.c line 63

Quoted 6 lines

mm/memory-tiers.c new file +5 -0

+
+static void memory_tier_device_release(struct device *dev)
+{
+	struct memory_tier *tier = to_memory_tier(dev);
+

Do we need some ref counts on memory_tier?
If there is another device still using the same memtier, free below could cause problem.

Quoted 4 lines

mm/memory-tiers.c new file +3 -0

+	kfree(tier);
+}
+

...

On 6/8/22 12:13 AM, Tim Chen wrote:
...

Quoted 11 lines

mm/memory-tiers.c line 63

Quoted 6 lines

mm/memory-tiers.c new file +5 -0

+
+static void memory_tier_device_release(struct device *dev)
+{
+	struct memory_tier *tier = to_memory_tier(dev);
+

Do we need some ref counts on memory_tier?
If there is another device still using the same memtier, free below could cause problem.

Quoted 5 lines

Quoted 4 lines

mm/memory-tiers.c new file +3 -0

+	kfree(tier);
+}
+

...

The lifecycle of the memory_tier struct is tied to the sysfs device life time. ie, memory_tier_device_relese get called only after the last reference on that sysfs dev object is released. Hence we can be sure there is no userspace that is keeping one of the memtier related sysfs file open.

W.r.t other memory device sharing the same memtier, we unregister the sysfs device only when the memory tier nodelist is empty. That is no memory device is present in this memory tier.

Quoted 26 lines · Aneesh Kumar K V

On 6/8/22 12:13 AM, Tim Chen wrote:
...

Quoted 11 lines

mm/memory-tiers.c line 63

Quoted 6 lines

mm/memory-tiers.c new file +5 -0

+
+static void memory_tier_device_release(struct device *dev)
+{
+	struct memory_tier *tier = to_memory_tier(dev);
+

Do we need some ref counts on memory_tier?
If there is another device still using the same memtier, free below could cause problem.

Quoted 5 lines

Quoted 4 lines

mm/memory-tiers.c new file +3 -0

+	kfree(tier);
+}
+

...

The lifecycle of the memory_tier struct is tied to the sysfs device life time. ie, memory_tier_device_relese get called only after the last reference on that sysfs dev object is released. Hence we can be sure there is no userspace that is keeping one of the memtier related sysfs file open.

Quoted 3 lines

W.r.t other memory device sharing the same memtier, we unregister the sysfs device only when the memory tier nodelist is empty. That is no memory device is present in this memory tier.

memory_tier isn't only used by user space. It is used inside kernel too. If some kernel code get a pointer to struct memory_tier, we need to guarantee the pointer will not be freed under us.

And as Tim pointed
out, we need to use it in hot path (for statistics), so some kind of rcu lock may be good.

In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.

This current memory tier kernel interface needs to be improved for several important use cases,

The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.

The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.

With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.

The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.

This patch series address the above by defining memory tiers explicitly.

This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at

/sys/devices/system/memtier/memtierN/

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier and memtier1 is the lowest tier.

The rank value of each memtier should be unique.

A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.

For now we are not adding the dynamic number of memory tiers. But a future series supporting that is possible. Currently number of tiers supported is limitted to MAX_MEMORY_TIERS(3). When doing memory hotplug, if not added to a memory tier, the NUMA node gets added to DEFAULT_MEMORY_TIER(1).

This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].

[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Suggested-by Wei Xu <[email protected]>

Signed-off-by Jagdish Gediya <[email protected]>

Signed-off-by Aneesh Kumar K.V <[email protected]>

---
 include/linux/memory-tiers.h |  20 ++++
 mm/Kconfig                   |  11 ++
 mm/Makefile                  |   1 +
 mm/memory-tiers.c            | 188 +++++++++++++++++++++++++++++++++++
 4 files changed, 220 insertions(+)
 create mode 100644 include/linux/memory-tiers.h
 create mode 100644 mm/memory-tiers.c

include/linux/memory-tiers.h new file +20 -0

new file mode 100644
index 000000000000..e17f6b4ee177
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU    0
+#define MEMORY_TIER_DRAM       1
+#define MEMORY_TIER_PMEM       2
+
+#define MEMORY_RANK_HBM_GPU    300
+#define MEMORY_RANK_DRAM       200
+#define MEMORY_RANK_PMEM       100
+
+#define DEFAULT_MEMORY_TIER    MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif

mm/Kconfig +0 -0

index 169e64192e48..08a3d330740b 100644
@@ -614,6 +614,17 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
 config ARCH_ENABLE_THP_MIGRATION
        bool
 

mm/Kconfig +1 -0

+
 config HUGETLB_PAGE_SIZE_VARIABLE
        def_bool n
        help

mm/Makefile +1 -0

index 6f9ffa968a1a..482557fbc9d1 100644
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
 obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_MEMTEST)          += memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
 obj-$(CONFIG_PAGE_COUNTER) += page_counter.o

mm/memory-tiers.c new file +188 -1

new file mode 100644
index 000000000000..7de18d94a08d
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/nodemask.h>
+#include <linux/slab.h>
+#include <linux/memory-tiers.h>
+
+struct memory_tier {
+       struct list_head list;
+       struct device dev;
+       nodemask_t nodelist;
+       int rank;
+};
+
+#define to_memory_tier(device) container_of(device, struct memory_tier, dev)
+
+static struct bus_type memory_tier_subsys = {
+       .name = "memtier",
+       .dev_name = "memtier",
+};
+
+static DEFINE_MUTEX(memory_tier_lock);
+static LIST_HEAD(memory_tiers);
+
+
+static ssize_t nodelist_show(struct device *dev,
+                            struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%*pbl\n",
+                         nodemask_pr_args(&memtier->nodelist));
+}
+static DEVICE_ATTR_RO(nodelist);
+
+static ssize_t rank_show(struct device *dev,
+                        struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%d\n", memtier->rank);
+}
+static DEVICE_ATTR_RO(rank);
+
+static struct attribute *memory_tier_dev_attrs[] = {
+       &dev_attr_nodelist.attr,
+       &dev_attr_rank.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_dev_group = {
+       .attrs = memory_tier_dev_attrs,
+};
+
+static const struct attribute_group *memory_tier_dev_groups[] = {
+       &memory_tier_dev_group,
+       NULL
+};
+
+static void memory_tier_device_release(struct device *dev)
+{
+       struct memory_tier *tier = to_memory_tier(dev);
+
+       kfree(tier);
+}
+
+/*
+ * Keep it simple by having  direct mapping between
+ * tier index and rank value.
+ */
+static inline int get_rank_from_tier(unsigned int tier)
+{
+       switch (tier) {
+       case MEMORY_TIER_HBM_GPU:
+               return MEMORY_RANK_HBM_GPU;
+       case MEMORY_TIER_DRAM:
+               return MEMORY_RANK_DRAM;
+       case MEMORY_TIER_PMEM:
+               return MEMORY_RANK_PMEM;
+       }
+
+       return 0;
+}
+
+static void insert_memory_tier(struct memory_tier *memtier)
+{
+       struct list_head *ent;
+       struct memory_tier *tmp_memtier;
+
+       list_for_each(ent, &memory_tiers) {
+               tmp_memtier = list_entry(ent, struct memory_tier, list);
+               if (tmp_memtier->rank < memtier->rank) {
+                       list_add_tail(&memtier->list, ent);
+                       return;
+               }
+       }
+       list_add_tail(&memtier->list, &memory_tiers);
+}
+
+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+       int error;
+       struct memory_tier *memtier;
+
+       if (tier >= MAX_MEMORY_TIERS)
+               return NULL;
+
+       memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+       if (!memtier)
+               return NULL;
+
+       memtier->dev.id = tier;
+       memtier->rank = get_rank_from_tier(tier);
+       memtier->dev.bus = &memory_tier_subsys;
+       memtier->dev.release = memory_tier_device_release;
+       memtier->dev.groups = memory_tier_dev_groups;
+
+       insert_memory_tier(memtier);
+
+       error = device_register(&memtier->dev);
+       if (error) {
+               list_del(&memtier->list);
+               put_device(&memtier->dev);
+               return NULL;
+       }
+       return memtier;
+}
+
+__maybe_unused // temporay to prevent warnings during bisects
+static void unregister_memory_tier(struct memory_tier *memtier)
+{
+       list_del(&memtier->list);
+       device_unregister(&memtier->dev);
+}
+
+static ssize_t
+max_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "%d\n", MAX_MEMORY_TIERS);
+}
+static DEVICE_ATTR_RO(max_tier);
+
+static ssize_t
+default_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "memtier%d\n", DEFAULT_MEMORY_TIER);
+}
+static DEVICE_ATTR_RO(default_tier);
+
+static struct attribute *memory_tier_attrs[] = {
+       &dev_attr_max_tier.attr,
+       &dev_attr_default_tier.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_attr_group = {
+       .attrs = memory_tier_attrs,
+};
+
+static const struct attribute_group *memory_tier_attr_groups[] = {
+       &memory_tier_attr_group,
+       NULL,
+};
+
+static int __init memory_tier_init(void)
+{
+       int ret;
+       struct memory_tier *memtier;
+
+       ret = subsys_system_register(&memory_tier_subsys, memory_tier_attr_groups);
+       if (ret)
+               panic("%s() failed to register subsystem: %d\n", __func__, ret);
+
+       /*
+        * Register only default memory tier to hide all empty
+        * memory tier from sysfs.
+        */
+       memtier = register_memory_tier(DEFAULT_MEMORY_TIER);
+       if (!memtier)
+               panic("%s() failed to register memory tier: %d\n", __func__, ret);
+
+       /* CPU only nodes are not part of memory tiers. */
+       memtier->nodelist = node_states[N_MEMORY];
+
+       return 0;
+}
+subsys_initcall(memory_tier_init);
+
--

2.36.1

Quoted 123 lines · Aneesh Kumar K.V

In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.

This current memory tier kernel interface needs to be improved for several important use cases,

The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.

The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.

With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.

The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.

This patch series address the above by defining memory tiers explicitly.

This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at

/sys/devices/system/memtier/memtierN/

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier and memtier1 is the lowest tier.

The rank value of each memtier should be unique.

A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.

For now we are not adding the dynamic number of memory tiers. But a future series supporting that is possible. Currently number of tiers supported is limitted to MAX_MEMORY_TIERS(3). When doing memory hotplug, if not added to a memory tier, the NUMA node gets added to DEFAULT_MEMORY_TIER(1).

This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].

[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Suggested-by Wei Xu <[email protected]>

Signed-off-by Jagdish Gediya <[email protected]>

Signed-off-by Aneesh Kumar K.V <[email protected]>

---

 include/linux/memory-tiers.h |  20 ++++
 mm/Kconfig                   |  11 ++
 mm/Makefile                  |   1 +
 mm/memory-tiers.c            | 188 +++++++++++++++++++++++++++++++++++
 4 files changed, 220 insertions(+)

 create mode 100644 include/linux/memory-tiers.h
 create mode 100644 mm/memory-tiers.c

include/linux/memory-tiers.h new file +20 -0

new file mode 100644
index 000000000000..e17f6b4ee177
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU    0
+#define MEMORY_TIER_DRAM       1
+#define MEMORY_TIER_PMEM       2
+
+#define MEMORY_RANK_HBM_GPU    300
+#define MEMORY_RANK_DRAM       200
+#define MEMORY_RANK_PMEM       100
+
+#define DEFAULT_MEMORY_TIER    MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif

mm/Kconfig +0 -0

index 169e64192e48..08a3d330740b 100644
@@ -614,6 +614,17 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION

 config ARCH_ENABLE_THP_MIGRATION
        bool

mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 626

Quoted 10 lines

mm/Kconfig +10 -0

+config TIERED_MEMORY
+       bool "Support for explicit memory tiers"
+       def_bool n
+       depends on MIGRATION && NUMA
+       help
+         Support to split nodes into memory tiers explicitly and
+         to demote pages on reclaim to lower tiers. This option
+         also exposes sysfs interface to read nodes available in
+         specific tier and to move specific node among different
+         possible tiers.

IMHO we should not need a new kernel config. If tiering is not present then there is just one tier on the system. And tiering is a kind of hardware configuration, the information could be shown regardless of whether demotion/promotion is supported/enabled or not.

Quoted 213 lines

+
 config HUGETLB_PAGE_SIZE_VARIABLE
        def_bool n
        help

mm/Makefile +0 -0

index 6f9ffa968a1a..482557fbc9d1 100644
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/

 obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_MEMTEST) += memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o

mm/memory-tiers.c new file +188 -1

new file mode 100644
index 000000000000..7de18d94a08d
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/nodemask.h>
+#include <linux/slab.h>
+#include <linux/memory-tiers.h>
+
+struct memory_tier {
+       struct list_head list;
+       struct device dev;
+       nodemask_t nodelist;
+       int rank;
+};
+
+#define to_memory_tier(device) container_of(device, struct memory_tier, dev)
+
+static struct bus_type memory_tier_subsys = {
+       .name = "memtier",
+       .dev_name = "memtier",
+};
+
+static DEFINE_MUTEX(memory_tier_lock);
+static LIST_HEAD(memory_tiers);
+
+
+static ssize_t nodelist_show(struct device *dev,
+                            struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%*pbl\n",
+                         nodemask_pr_args(&memtier->nodelist));
+}
+static DEVICE_ATTR_RO(nodelist);
+
+static ssize_t rank_show(struct device *dev,
+                        struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%d\n", memtier->rank);
+}
+static DEVICE_ATTR_RO(rank);
+
+static struct attribute *memory_tier_dev_attrs[] = {
+       &dev_attr_nodelist.attr,
+       &dev_attr_rank.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_dev_group = {
+       .attrs = memory_tier_dev_attrs,
+};
+
+static const struct attribute_group *memory_tier_dev_groups[] = {
+       &memory_tier_dev_group,
+       NULL
+};
+
+static void memory_tier_device_release(struct device *dev)
+{
+       struct memory_tier *tier = to_memory_tier(dev);
+
+       kfree(tier);
+}
+
+/*
+ * Keep it simple by having  direct mapping between
+ * tier index and rank value.
+ */
+static inline int get_rank_from_tier(unsigned int tier)
+{
+       switch (tier) {
+       case MEMORY_TIER_HBM_GPU:
+               return MEMORY_RANK_HBM_GPU;
+       case MEMORY_TIER_DRAM:
+               return MEMORY_RANK_DRAM;
+       case MEMORY_TIER_PMEM:
+               return MEMORY_RANK_PMEM;
+       }
+
+       return 0;
+}
+
+static void insert_memory_tier(struct memory_tier *memtier)
+{
+       struct list_head *ent;
+       struct memory_tier *tmp_memtier;
+
+       list_for_each(ent, &memory_tiers) {
+               tmp_memtier = list_entry(ent, struct memory_tier, list);
+               if (tmp_memtier->rank < memtier->rank) {
+                       list_add_tail(&memtier->list, ent);
+                       return;
+               }
+       }
+       list_add_tail(&memtier->list, &memory_tiers);
+}
+
+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+       int error;
+       struct memory_tier *memtier;
+
+       if (tier >= MAX_MEMORY_TIERS)
+               return NULL;
+
+       memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+       if (!memtier)
+               return NULL;
+
+       memtier->dev.id = tier;
+       memtier->rank = get_rank_from_tier(tier);
+       memtier->dev.bus = &memory_tier_subsys;
+       memtier->dev.release = memory_tier_device_release;
+       memtier->dev.groups = memory_tier_dev_groups;
+
+       insert_memory_tier(memtier);
+
+       error = device_register(&memtier->dev);
+       if (error) {
+               list_del(&memtier->list);
+               put_device(&memtier->dev);
+               return NULL;
+       }
+       return memtier;
+}
+
+__maybe_unused // temporay to prevent warnings during bisects
+static void unregister_memory_tier(struct memory_tier *memtier)
+{
+       list_del(&memtier->list);
+       device_unregister(&memtier->dev);
+}
+
+static ssize_t
+max_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "%d\n", MAX_MEMORY_TIERS);
+}
+static DEVICE_ATTR_RO(max_tier);
+
+static ssize_t
+default_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "memtier%d\n", DEFAULT_MEMORY_TIER);
+}
+static DEVICE_ATTR_RO(default_tier);
+
+static struct attribute *memory_tier_attrs[] = {
+       &dev_attr_max_tier.attr,
+       &dev_attr_default_tier.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_attr_group = {
+       .attrs = memory_tier_attrs,
+};
+
+static const struct attribute_group *memory_tier_attr_groups[] = {
+       &memory_tier_attr_group,
+       NULL,
+};
+
+static int __init memory_tier_init(void)
+{
+       int ret;
+       struct memory_tier *memtier;
+
+       ret = subsys_system_register(&memory_tier_subsys, memory_tier_attr_groups);
+       if (ret)
+               panic("%s() failed to register subsystem: %d\n", __func__, ret);
+
+       /*
+        * Register only default memory tier to hide all empty
+        * memory tier from sysfs.
+        */
+       memtier = register_memory_tier(DEFAULT_MEMORY_TIER);
+       if (!memtier)
+               panic("%s() failed to register memory tier: %d\n", __func__, ret);
+
+       /* CPU only nodes are not part of memory tiers. */
+       memtier->nodelist = node_states[N_MEMORY];
+
+       return 0;
+}
+subsys_initcall(memory_tier_init);
+
--

2.36.1

Quoted 125 lines · Yang Shi

Quoted 123 lines · Aneesh Kumar K.V

In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.

This current memory tier kernel interface needs to be improved for several important use cases,

The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.

The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.

With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.

The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.

This patch series address the above by defining memory tiers explicitly.

This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at

/sys/devices/system/memtier/memtierN/

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier and memtier1 is the lowest tier.

The rank value of each memtier should be unique.

A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.

For now we are not adding the dynamic number of memory tiers. But a future series supporting that is possible. Currently number of tiers supported is limitted to MAX_MEMORY_TIERS(3). When doing memory hotplug, if not added to a memory tier, the NUMA node gets added to DEFAULT_MEMORY_TIER(1).

This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].

[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Suggested-by Wei Xu <[email protected]>

Signed-off-by Jagdish Gediya <[email protected]>

Signed-off-by Aneesh Kumar K.V <[email protected]>

---
 include/linux/memory-tiers.h |  20 ++++
 mm/Kconfig                   |  11 ++
 mm/Makefile                  |   1 +
 mm/memory-tiers.c            | 188 +++++++++++++++++++++++++++++++++++
 4 files changed, 220 insertions(+)
 create mode 100644 include/linux/memory-tiers.h
 create mode 100644 mm/memory-tiers.c

include/linux/memory-tiers.h new file +20 -0

new file mode 100644
index 000000000000..e17f6b4ee177
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU    0
+#define MEMORY_TIER_DRAM       1
+#define MEMORY_TIER_PMEM       2
+
+#define MEMORY_RANK_HBM_GPU    300
+#define MEMORY_RANK_DRAM       200
+#define MEMORY_RANK_PMEM       100
+
+#define DEFAULT_MEMORY_TIER    MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif

mm/Kconfig +0 -0

index 169e64192e48..08a3d330740b 100644
@@ -614,6 +614,17 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
 config ARCH_ENABLE_THP_MIGRATION
        bool
 

Quoted 15 lines

mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 626

Quoted 10 lines

mm/Kconfig +10 -0

+config TIERED_MEMORY
+       bool "Support for explicit memory tiers"
+       def_bool n
+       depends on MIGRATION && NUMA
+       help
+         Support to split nodes into memory tiers explicitly and
+         to demote pages on reclaim to lower tiers. This option
+         also exposes sysfs interface to read nodes available in
+         specific tier and to move specific node among different
+         possible tiers.

IMHO we should not need a new kernel config. If tiering is not present then there is just one tier on the system. And tiering is a kind of hardware configuration, the information could be shown regardless of whether demotion/promotion is supported/enabled or not.

I think so too. At least it appears unnecessary to let the user turn on/off it at configuration time.

All the code should be enclosed by #if defined(CONFIG_NUMA) && defined(CONIFIG_MIGRATION). So we will not waste memory in small systems.

Best Regards,
Huang, Ying

Quoted 213 lines

Quoted 213 lines

mm/Kconfig +1 -0

+
 config HUGETLB_PAGE_SIZE_VARIABLE
        def_bool n
        help

mm/Makefile +1 -0

index 6f9ffa968a1a..482557fbc9d1 100644
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
 obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_MEMTEST)          += memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
 obj-$(CONFIG_PAGE_COUNTER) += page_counter.o

mm/memory-tiers.c new file +188 -1

new file mode 100644
index 000000000000..7de18d94a08d
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/nodemask.h>
+#include <linux/slab.h>
+#include <linux/memory-tiers.h>
+
+struct memory_tier {
+       struct list_head list;
+       struct device dev;
+       nodemask_t nodelist;
+       int rank;
+};
+
+#define to_memory_tier(device) container_of(device, struct memory_tier, dev)
+
+static struct bus_type memory_tier_subsys = {
+       .name = "memtier",
+       .dev_name = "memtier",
+};
+
+static DEFINE_MUTEX(memory_tier_lock);
+static LIST_HEAD(memory_tiers);
+
+
+static ssize_t nodelist_show(struct device *dev,
+                            struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%*pbl\n",
+                         nodemask_pr_args(&memtier->nodelist));
+}
+static DEVICE_ATTR_RO(nodelist);
+
+static ssize_t rank_show(struct device *dev,
+                        struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%d\n", memtier->rank);
+}
+static DEVICE_ATTR_RO(rank);
+
+static struct attribute *memory_tier_dev_attrs[] = {
+       &dev_attr_nodelist.attr,
+       &dev_attr_rank.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_dev_group = {
+       .attrs = memory_tier_dev_attrs,
+};
+
+static const struct attribute_group *memory_tier_dev_groups[] = {
+       &memory_tier_dev_group,
+       NULL
+};
+
+static void memory_tier_device_release(struct device *dev)
+{
+       struct memory_tier *tier = to_memory_tier(dev);
+
+       kfree(tier);
+}
+
+/*
+ * Keep it simple by having  direct mapping between
+ * tier index and rank value.
+ */
+static inline int get_rank_from_tier(unsigned int tier)
+{
+       switch (tier) {
+       case MEMORY_TIER_HBM_GPU:
+               return MEMORY_RANK_HBM_GPU;
+       case MEMORY_TIER_DRAM:
+               return MEMORY_RANK_DRAM;
+       case MEMORY_TIER_PMEM:
+               return MEMORY_RANK_PMEM;
+       }
+
+       return 0;
+}
+
+static void insert_memory_tier(struct memory_tier *memtier)
+{
+       struct list_head *ent;
+       struct memory_tier *tmp_memtier;
+
+       list_for_each(ent, &memory_tiers) {
+               tmp_memtier = list_entry(ent, struct memory_tier, list);
+               if (tmp_memtier->rank < memtier->rank) {
+                       list_add_tail(&memtier->list, ent);
+                       return;
+               }
+       }
+       list_add_tail(&memtier->list, &memory_tiers);
+}
+
+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+       int error;
+       struct memory_tier *memtier;
+
+       if (tier >= MAX_MEMORY_TIERS)
+               return NULL;
+
+       memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+       if (!memtier)
+               return NULL;
+
+       memtier->dev.id = tier;
+       memtier->rank = get_rank_from_tier(tier);
+       memtier->dev.bus = &memory_tier_subsys;
+       memtier->dev.release = memory_tier_device_release;
+       memtier->dev.groups = memory_tier_dev_groups;
+
+       insert_memory_tier(memtier);
+
+       error = device_register(&memtier->dev);
+       if (error) {
+               list_del(&memtier->list);
+               put_device(&memtier->dev);
+               return NULL;
+       }
+       return memtier;
+}
+
+__maybe_unused // temporay to prevent warnings during bisects
+static void unregister_memory_tier(struct memory_tier *memtier)
+{
+       list_del(&memtier->list);
+       device_unregister(&memtier->dev);
+}
+
+static ssize_t
+max_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "%d\n", MAX_MEMORY_TIERS);
+}
+static DEVICE_ATTR_RO(max_tier);
+
+static ssize_t
+default_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "memtier%d\n", DEFAULT_MEMORY_TIER);
+}
+static DEVICE_ATTR_RO(default_tier);
+
+static struct attribute *memory_tier_attrs[] = {
+       &dev_attr_max_tier.attr,
+       &dev_attr_default_tier.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_attr_group = {
+       .attrs = memory_tier_attrs,
+};
+
+static const struct attribute_group *memory_tier_attr_groups[] = {
+       &memory_tier_attr_group,
+       NULL,
+};
+
+static int __init memory_tier_init(void)
+{
+       int ret;
+       struct memory_tier *memtier;
+
+       ret = subsys_system_register(&memory_tier_subsys, memory_tier_attr_groups);
+       if (ret)
+               panic("%s() failed to register subsystem: %d\n", __func__, ret);
+
+       /*
+        * Register only default memory tier to hide all empty
+        * memory tier from sysfs.
+        */
+       memtier = register_memory_tier(DEFAULT_MEMORY_TIER);
+       if (!memtier)
+               panic("%s() failed to register memory tier: %d\n", __func__, ret);
+
+       /* CPU only nodes are not part of memory tiers. */
+       memtier->nodelist = node_states[N_MEMORY];
+
+       return 0;
+}
+subsys_initcall(memory_tier_init);
+
--

2.36.1

Quoted 146 lines · Ying Huang

Quoted 125 lines · Yang Shi

Quoted 123 lines · Aneesh Kumar K.V

In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.

This current memory tier kernel interface needs to be improved for several important use cases,

The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.

The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.

With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.

The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.

This patch series address the above by defining memory tiers explicitly.

This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at

/sys/devices/system/memtier/memtierN/

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier and memtier1 is the lowest tier.

The rank value of each memtier should be unique.

A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.

For now we are not adding the dynamic number of memory tiers. But a future series supporting that is possible. Currently number of tiers supported is limitted to MAX_MEMORY_TIERS(3). When doing memory hotplug, if not added to a memory tier, the NUMA node gets added to DEFAULT_MEMORY_TIER(1).

This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].

[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Suggested-by Wei Xu <[email protected]>

Signed-off-by Jagdish Gediya <[email protected]>

Signed-off-by Aneesh Kumar K.V <[email protected]>

---

 include/linux/memory-tiers.h |  20 ++++
 mm/Kconfig                   |  11 ++
 mm/Makefile                  |   1 +
 mm/memory-tiers.c            | 188 +++++++++++++++++++++++++++++++++++
 4 files changed, 220 insertions(+)

 create mode 100644 include/linux/memory-tiers.h
 create mode 100644 mm/memory-tiers.c

include/linux/memory-tiers.h new file +20 -0

new file mode 100644
index 000000000000..e17f6b4ee177
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU    0
+#define MEMORY_TIER_DRAM       1
+#define MEMORY_TIER_PMEM       2
+
+#define MEMORY_RANK_HBM_GPU    300
+#define MEMORY_RANK_DRAM       200
+#define MEMORY_RANK_PMEM       100
+
+#define DEFAULT_MEMORY_TIER    MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif

mm/Kconfig +0 -0

index 169e64192e48..08a3d330740b 100644
@@ -614,6 +614,17 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION

 config ARCH_ENABLE_THP_MIGRATION
        bool

Quoted 15 lines

mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 626

Quoted 10 lines

mm/Kconfig +10 -0

+config TIERED_MEMORY
+       bool "Support for explicit memory tiers"
+       def_bool n
+       depends on MIGRATION && NUMA
+       help
+         Support to split nodes into memory tiers explicitly and
+         to demote pages on reclaim to lower tiers. This option
+         also exposes sysfs interface to read nodes available in
+         specific tier and to move specific node among different
+         possible tiers.

IMHO we should not need a new kernel config. If tiering is not present then there is just one tier on the system. And tiering is a kind of hardware configuration, the information could be shown regardless of whether demotion/promotion is supported/enabled or not.

I think so too. At least it appears unnecessary to let the user turn on/off it at configuration time.

Quoted 3 lines

All the code should be enclosed by #if defined(CONFIG_NUMA) && defined(CONIFIG_MIGRATION). So we will not waste memory in small systems.

CONFIG_NUMA alone should be good enough. CONFIG_MIGRATION is enabled by default if NUMA is enabled. And MIGRATION is just used to support demotion/promotion. Memory tiers exist even though demotion/promotion is not supported, right?

Quoted 216 lines

Quoted 214 lines

Quoted 214 lines

+
 config HUGETLB_PAGE_SIZE_VARIABLE
        def_bool n
        help

mm/Makefile +0 -0

index 6f9ffa968a1a..482557fbc9d1 100644
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/

 obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_MEMTEST) += memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o

mm/memory-tiers.c new file +188 -1

new file mode 100644
index 000000000000..7de18d94a08d
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/nodemask.h>
+#include <linux/slab.h>
+#include <linux/memory-tiers.h>
+
+struct memory_tier {
+       struct list_head list;
+       struct device dev;
+       nodemask_t nodelist;
+       int rank;
+};
+
+#define to_memory_tier(device) container_of(device, struct memory_tier, dev)
+
+static struct bus_type memory_tier_subsys = {
+       .name = "memtier",
+       .dev_name = "memtier",
+};
+
+static DEFINE_MUTEX(memory_tier_lock);
+static LIST_HEAD(memory_tiers);
+
+
+static ssize_t nodelist_show(struct device *dev,
+                            struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%*pbl\n",
+                         nodemask_pr_args(&memtier->nodelist));
+}
+static DEVICE_ATTR_RO(nodelist);
+
+static ssize_t rank_show(struct device *dev,
+                        struct device_attribute *attr, char *buf)
+{
+       struct memory_tier *memtier = to_memory_tier(dev);
+
+       return sysfs_emit(buf, "%d\n", memtier->rank);
+}
+static DEVICE_ATTR_RO(rank);
+
+static struct attribute *memory_tier_dev_attrs[] = {
+       &dev_attr_nodelist.attr,
+       &dev_attr_rank.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_dev_group = {
+       .attrs = memory_tier_dev_attrs,
+};
+
+static const struct attribute_group *memory_tier_dev_groups[] = {
+       &memory_tier_dev_group,
+       NULL
+};
+
+static void memory_tier_device_release(struct device *dev)
+{
+       struct memory_tier *tier = to_memory_tier(dev);
+
+       kfree(tier);
+}
+
+/*
+ * Keep it simple by having  direct mapping between
+ * tier index and rank value.
+ */
+static inline int get_rank_from_tier(unsigned int tier)
+{
+       switch (tier) {
+       case MEMORY_TIER_HBM_GPU:
+               return MEMORY_RANK_HBM_GPU;
+       case MEMORY_TIER_DRAM:
+               return MEMORY_RANK_DRAM;
+       case MEMORY_TIER_PMEM:
+               return MEMORY_RANK_PMEM;
+       }
+
+       return 0;
+}
+
+static void insert_memory_tier(struct memory_tier *memtier)
+{
+       struct list_head *ent;
+       struct memory_tier *tmp_memtier;
+
+       list_for_each(ent, &memory_tiers) {
+               tmp_memtier = list_entry(ent, struct memory_tier, list);
+               if (tmp_memtier->rank < memtier->rank) {
+                       list_add_tail(&memtier->list, ent);
+                       return;
+               }
+       }
+       list_add_tail(&memtier->list, &memory_tiers);
+}
+
+static struct memory_tier *register_memory_tier(unsigned int tier)
+{
+       int error;
+       struct memory_tier *memtier;
+
+       if (tier >= MAX_MEMORY_TIERS)
+               return NULL;
+
+       memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+       if (!memtier)
+               return NULL;
+
+       memtier->dev.id = tier;
+       memtier->rank = get_rank_from_tier(tier);
+       memtier->dev.bus = &memory_tier_subsys;
+       memtier->dev.release = memory_tier_device_release;
+       memtier->dev.groups = memory_tier_dev_groups;
+
+       insert_memory_tier(memtier);
+
+       error = device_register(&memtier->dev);
+       if (error) {
+               list_del(&memtier->list);
+               put_device(&memtier->dev);
+               return NULL;
+       }
+       return memtier;
+}
+
+__maybe_unused // temporay to prevent warnings during bisects
+static void unregister_memory_tier(struct memory_tier *memtier)
+{
+       list_del(&memtier->list);
+       device_unregister(&memtier->dev);
+}
+
+static ssize_t
+max_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "%d\n", MAX_MEMORY_TIERS);
+}
+static DEVICE_ATTR_RO(max_tier);
+
+static ssize_t
+default_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       return sysfs_emit(buf, "memtier%d\n", DEFAULT_MEMORY_TIER);
+}
+static DEVICE_ATTR_RO(default_tier);
+
+static struct attribute *memory_tier_attrs[] = {
+       &dev_attr_max_tier.attr,
+       &dev_attr_default_tier.attr,
+       NULL
+};
+
+static const struct attribute_group memory_tier_attr_group = {
+       .attrs = memory_tier_attrs,
+};
+
+static const struct attribute_group *memory_tier_attr_groups[] = {
+       &memory_tier_attr_group,
+       NULL,
+};
+
+static int __init memory_tier_init(void)
+{
+       int ret;
+       struct memory_tier *memtier;
+
+       ret = subsys_system_register(&memory_tier_subsys, memory_tier_attr_groups);
+       if (ret)
+               panic("%s() failed to register subsystem: %d\n", __func__, ret);
+
+       /*
+        * Register only default memory tier to hide all empty
+        * memory tier from sysfs.
+        */
+       memtier = register_memory_tier(DEFAULT_MEMORY_TIER);
+       if (!memtier)
+               panic("%s() failed to register memory tier: %d\n", __func__, ret);
+
+       /* CPU only nodes are not part of memory tiers. */
+       memtier->nodelist = node_states[N_MEMORY];
+
+       return 0;
+}
+subsys_initcall(memory_tier_init);
+
--

2.36.1

Quoted 123 lines · Aneesh Kumar K.V

In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.

This current memory tier kernel interface needs to be improved for several important use cases,

The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.

The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.

With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.

The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.

This patch series address the above by defining memory tiers explicitly.

This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at

/sys/devices/system/memtier/memtierN/

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier and memtier1 is the lowest tier.

The rank value of each memtier should be unique.

A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.

For now we are not adding the dynamic number of memory tiers. But a future series supporting that is possible. Currently number of tiers supported is limitted to MAX_MEMORY_TIERS(3). When doing memory hotplug, if not added to a memory tier, the NUMA node gets added to DEFAULT_MEMORY_TIER(1).

This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].

[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Suggested-by Wei Xu <[email protected]>

Signed-off-by Jagdish Gediya <[email protected]>

Signed-off-by Aneesh Kumar K.V <[email protected]>

---
  include/linux/memory-tiers.h |  20 ++++
  mm/Kconfig                   |  11 ++
  mm/Makefile                  |   1 +
  mm/memory-tiers.c            | 188 +++++++++++++++++++++++++++++++++++
  4 files changed, 220 insertions(+)

create mode 100644 include/linux/memory-tiers.h
create mode 100644 mm/memory-tiers.c

include/linux/memory-tiers.h new file +20 -0

new file mode 100644
index 000000000000..e17f6b4ee177
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU    0
+#define MEMORY_TIER_DRAM       1
+#define MEMORY_TIER_PMEM       2
+
+#define MEMORY_RANK_HBM_GPU    300
+#define MEMORY_RANK_DRAM       200
+#define MEMORY_RANK_PMEM       100
+
+#define DEFAULT_MEMORY_TIER    MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif

mm/Kconfig +0 -0

index 169e64192e48..08a3d330740b 100644
@@ -614,6 +614,17 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
  config ARCH_ENABLE_THP_MIGRATION
         bool
 

mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 626

Quoted 10 lines

mm/Kconfig +10 -0

+config TIERED_MEMORY
+       bool "Support for explicit memory tiers"
+       def_bool n
+       depends on MIGRATION && NUMA
+       help
+         Support to split nodes into memory tiers explicitly and
+         to demote pages on reclaim to lower tiers. This option
+         also exposes sysfs interface to read nodes available in
+         specific tier and to move specific node among different
+         possible tiers.

IMHO we should not need a new kernel config. If tiering is not present then there is just one tier on the system. And tiering is a kind of hardware configuration, the information could be shown regardless of whether demotion/promotion is supported/enabled or not.

Quoted 125 lines · Yang Shi

Quoted 123 lines · Aneesh Kumar K.V

In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.

This current memory tier kernel interface needs to be improved for several important use cases,

The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.

The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.

With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.

The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.

This patch series address the above by defining memory tiers explicitly.

This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at

/sys/devices/system/memtier/memtierN/

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier and memtier1 is the lowest tier.

The rank value of each memtier should be unique.

A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.

For now we are not adding the dynamic number of memory tiers. But a future series supporting that is possible. Currently number of tiers supported is limitted to MAX_MEMORY_TIERS(3). When doing memory hotplug, if not added to a memory tier, the NUMA node gets added to DEFAULT_MEMORY_TIER(1).

This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].

[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Suggested-by Wei Xu <[email protected]>

Signed-off-by Jagdish Gediya <[email protected]>

Signed-off-by Aneesh Kumar K.V <[email protected]>

---

  include/linux/memory-tiers.h |  20 ++++
  mm/Kconfig                   |  11 ++
  mm/Makefile                  |   1 +
  mm/memory-tiers.c            | 188 +++++++++++++++++++++++++++++++++++
  4 files changed, 220 insertions(+)

  create mode 100644 include/linux/memory-tiers.h
  create mode 100644 mm/memory-tiers.c

include/linux/memory-tiers.h new file +20 -0

new file mode 100644
index 000000000000..e17f6b4ee177
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU    0
+#define MEMORY_TIER_DRAM       1
+#define MEMORY_TIER_PMEM       2
+
+#define MEMORY_RANK_HBM_GPU    300
+#define MEMORY_RANK_DRAM       200
+#define MEMORY_RANK_PMEM       100
+
+#define DEFAULT_MEMORY_TIER    MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif

mm/Kconfig +0 -0

index 169e64192e48..08a3d330740b 100644
@@ -614,6 +614,17 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION

  config ARCH_ENABLE_THP_MIGRATION
         bool

Quoted 16 lines

mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 626

Quoted 10 lines

mm/Kconfig +10 -0

+config TIERED_MEMORY
+       bool "Support for explicit memory tiers"
+       def_bool n
+       depends on MIGRATION && NUMA
+       help
+         Support to split nodes into memory tiers explicitly and
+         to demote pages on reclaim to lower tiers. This option
+         also exposes sysfs interface to read nodes available in
+         specific tier and to move specific node among different
+         possible tiers.

IMHO we should not need a new kernel config. If tiering is not present then there is just one tier on the system. And tiering is a kind of hardware configuration, the information could be shown regardless of whether demotion/promotion is supported/enabled or not.

This was added so that we could avoid doing multiple

#if defined(CONFIG_MIGRATION) && defined(CONFIG_NUMA)

Initially I had that as def_bool y and depends on MIGRATION && NUMA. But it was later suggested that def_bool is not recommended for newer config.

How about

  config TIERED_MEMORY
   bool "Support for explicit memory tiers"

(quoted diff) +1 -8

-	def_bool n
-	depends on MIGRATION && NUMA
-	help
-	  Support to split nodes into memory tiers explicitly and
-	  to demote pages on reclaim to lower tiers. This option
-	  also exposes sysfs interface to read nodes available in
-	  specific tier and to move specific node among different
-	  possible tiers.
+	def_bool MIGRATION && NUMA

  config HUGETLB_PAGE_SIZE_VARIABLE
   def_bool n

ie, we just make it a Kconfig variable without exposing it to the user?

-aneesh

Quoted 125 lines · Yang Shi

Quoted 123 lines · Aneesh Kumar K.V

In the current kernel, memory tiers are defined implicitly via a demotion path relationship between NUMA nodes, which is created during the kernel initialization and updated when a NUMA node is hot-added or hot-removed. The current implementation puts all nodes with CPU into the top tier, and builds the tier hierarchy tier-by-tier by establishing the per-node demotion targets based on the distances between nodes.

This current memory tier kernel interface needs to be improved for several important use cases,

The current tier initialization code always initializes each memory-only NUMA node into a lower tier. But a memory-only NUMA node may have a high performance memory device (e.g. a DRAM device attached via CXL.mem or a DRAM-backed memory-only node on a virtual machine) and should be put into a higher tier.

The current tier hierarchy always puts CPU nodes into the top tier. But on a system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices should be in the top tier, and DRAM nodes with CPUs are better to be placed into the next lower tier.

With current kernel higher tier node can only be demoted to selected nodes on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order does not work in all use cases (e.g. some use cases may want to allow cross-socket demotion to another node in the same demotion tier as a fallback when the preferred demotion node is out of space), This demotion order is also inconsistent with the page allocation fallback order when all the nodes in a higher tier are out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.

The current kernel also don't provide any interfaces for the userspace to learn about the memory tier hierarchy in order to optimize its memory allocations.

This patch series address the above by defining memory tiers explicitly.

This patch introduce explicity memory tiers with ranks. The rank value of a memory tier is used to derive the demotion order between NUMA nodes. The memory tiers present in a system can be found at

/sys/devices/system/memtier/memtierN/

The nodes which are part of a specific memory tier can be listed via
/sys/devices/system/memtier/memtierN/nodelist

"Rank" is an opaque value. Its absolute value doesn't have any special meaning. But the rank values of different memtiers can be compared with each other to determine the memory tier order.

For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier and memtier1 is the lowest tier.

The rank value of each memtier should be unique.

A higher rank memory tier will appear first in the demotion order than a lower rank memory tier. ie. while reclaim we choose a node in higher rank memory tier to demote pages to as compared to a node in a lower rank memory tier.

For now we are not adding the dynamic number of memory tiers. But a future series supporting that is possible. Currently number of tiers supported is limitted to MAX_MEMORY_TIERS(3). When doing memory hotplug, if not added to a memory tier, the NUMA node gets added to DEFAULT_MEMORY_TIER(1).

This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].

[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Suggested-by Wei Xu <[email protected]>

Signed-off-by Jagdish Gediya <[email protected]>

Signed-off-by Aneesh Kumar K.V <[email protected]>

---
  include/linux/memory-tiers.h |  20 ++++
  mm/Kconfig                   |  11 ++
  mm/Makefile                  |   1 +
  mm/memory-tiers.c            | 188 +++++++++++++++++++++++++++++++++++
  4 files changed, 220 insertions(+)

create mode 100644 include/linux/memory-tiers.h
create mode 100644 mm/memory-tiers.c

include/linux/memory-tiers.h new file +20 -0

new file mode 100644
index 000000000000..e17f6b4ee177
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU    0
+#define MEMORY_TIER_DRAM       1
+#define MEMORY_TIER_PMEM       2
+
+#define MEMORY_RANK_HBM_GPU    300
+#define MEMORY_RANK_DRAM       200
+#define MEMORY_RANK_PMEM       100
+
+#define DEFAULT_MEMORY_TIER    MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif

mm/Kconfig +0 -0

index 169e64192e48..08a3d330740b 100644
@@ -614,6 +614,17 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
  config ARCH_ENABLE_THP_MIGRATION
         bool
 

Quoted 16 lines

mm/Kconfig config ARCH_ENABLE_HUGEPAGE_MIGRATION · line 626

Quoted 10 lines

mm/Kconfig +10 -0

+config TIERED_MEMORY
+       bool "Support for explicit memory tiers"
+       def_bool n
+       depends on MIGRATION && NUMA
+       help
+         Support to split nodes into memory tiers explicitly and
+         to demote pages on reclaim to lower tiers. This option
+         also exposes sysfs interface to read nodes available in
+         specific tier and to move specific node among different
+         possible tiers.

IMHO we should not need a new kernel config. If tiering is not present then there is just one tier on the system. And tiering is a kind of hardware configuration, the information could be shown regardless of whether demotion/promotion is supported/enabled or not.

This was added so that we could avoid doing multiple

#if defined(CONFIG_MIGRATION) && defined(CONFIG_NUMA)

Initially I had that as def_bool y and depends on MIGRATION && NUMA. But it was later suggested that def_bool is not recommended for newer config.

How about

config TIERED_MEMORY
bool "Support for explicit memory tiers"

config HUGETLB_PAGE_SIZE_VARIABLE
def_bool n

ie, we just make it a Kconfig variable without exposing it to the user?

-aneesh

On Tue, Jun 7, 2022 at 9:58 PM Aneesh Kumar K V
<[email protected]> wrote:

Review

Quoted 11 lines

config TIERED_MEMORY
bool "Support for explicit memory tiers"

(quoted diff) +1 -8

-       def_bool n
-       depends on MIGRATION && NUMA
-       help
-         Support to split nodes into memory tiers explicitly and
-         to demote pages on reclaim to lower tiers. This option
-         also exposes sysfs interface to read nodes available in
-         specific tier and to move specific node among different
-         possible tiers.
+       def_bool MIGRATION && NUMA

CONFIG_NUMA should be good enough. Memory tiering doesn't have to mean demotion/promotion has to be supported IMHO.

Quoted 6 lines

config HUGETLB_PAGE_SIZE_VARIABLE
def_bool n

ie, we just make it a Kconfig variable without exposing it to the user?

Quoted 2 lines · Yang Shi

On Tue, Jun 7, 2022 at 9:58 PM Aneesh Kumar K V
<[email protected]> wrote:

....

Quoted 19 lines

Review

Quoted 11 lines

config TIERED_MEMORY
bool "Support for explicit memory tiers"

(quoted diff) +1 -8

-       def_bool n
-       depends on MIGRATION && NUMA
-       help
-         Support to split nodes into memory tiers explicitly and
-         to demote pages on reclaim to lower tiers. This option
-         also exposes sysfs interface to read nodes available in
-         specific tier and to move specific node among different
-         possible tiers.
+       def_bool MIGRATION && NUMA

CONFIG_NUMA should be good enough. Memory tiering doesn't have to mean demotion/promotion has to be supported IMHO.

Quoted 4 lines

config HUGETLB_PAGE_SIZE_VARIABLE
def_bool n

Quoted 2 lines

Quoted 2 lines

ie, we just make it a Kconfig variable without exposing it to the user?

We can do that but that would also mean in order to avoid building the demotion targets etc we will now have to have multiple #ifdef CONFIG_MIGRATION in mm/memory-tiers.c . It builds without those #ifdef So these are not really build errors, but rather we will be building all the demotion targets for no real use with them.

What usecase do you have to expose memory tiers on a system with CONFIG_MIGRATION disabled? CONFIG_MIGRATION gets enabled in almost all configs these days due to its dependency against COMPACTION and TRANSPARENT_HUGEPAGE.

Unless there is a real need, I am wondering if we can avoid sprinkling #ifdef CONFIG_MIGRATION in mm/memory-tiers.c

-aneesh

include/linux/memory-tiers.h new file +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

Hi Aneesh,

Quoted 7 lines · Aneesh Kumar K.V

include/linux/memory-tiers.h new file +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

include/linux/memory-tiers.h line 16

Quoted 10 lines

include/linux/memory-tiers.h new file +10 -0

+#define MEMORY_TIER_HBM_GPU	0
+#define MEMORY_TIER_DRAM	1
+#define MEMORY_TIER_PMEM	2
+
+#define MEMORY_RANK_HBM_GPU	300
+#define MEMORY_RANK_DRAM	200
+#define MEMORY_RANK_PMEM	100
+
+#define DEFAULT_MEMORY_TIER	MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3

I understand the names are somewhat arbitrary, and the tier ID space can be expanded down the line by bumping MAX_MEMORY_TIERS.

But starting out with a packed ID space can get quite awkward for users when new tiers - especially intermediate tiers - show up in existing configurations. I mentioned in the other email that DRAM != DRAM, so new tiers seem inevitable already.

It could make sense to start with a bigger address space and spread out the list of kernel default tiers a bit within it:

MEMORY_TIER_GPU 0
MEMORY_TIER_DRAM 10
MEMORY_TIER_PMEM 20

Quoted 7 lines · Aneesh Kumar K.V

include/linux/memory-tiers.h new file +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

include/linux/memory-tiers.h line 16

Quoted 10 lines

include/linux/memory-tiers.h new file +10 -0

+#define MEMORY_TIER_HBM_GPU	0
+#define MEMORY_TIER_DRAM	1
+#define MEMORY_TIER_PMEM	2
+
+#define MEMORY_RANK_HBM_GPU	300
+#define MEMORY_RANK_DRAM	200
+#define MEMORY_RANK_PMEM	100
+
+#define DEFAULT_MEMORY_TIER	MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3

I understand the names are somewhat arbitrary, and the tier ID space can be expanded down the line by bumping MAX_MEMORY_TIERS.

But starting out with a packed ID space can get quite awkward for users when new tiers - especially intermediate tiers - show up in existing configurations. I mentioned in the other email that DRAM != DRAM, so new tiers seem inevitable already.

It could make sense to start with a bigger address space and spread out the list of kernel default tiers a bit within it:

MEMORY_TIER_GPU 0
MEMORY_TIER_DRAM 10
MEMORY_TIER_PMEM 20

Hello,

Quoted 27 lines · Johannes Weiner

Quoted 7 lines · Aneesh Kumar K.V

include/linux/memory-tiers.h new file +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

include/linux/memory-tiers.h line 16

Quoted 10 lines

include/linux/memory-tiers.h new file +10 -0

+#define MEMORY_TIER_HBM_GPU	0
+#define MEMORY_TIER_DRAM	1
+#define MEMORY_TIER_PMEM	2
+
+#define MEMORY_RANK_HBM_GPU	300
+#define MEMORY_RANK_DRAM	200
+#define MEMORY_RANK_PMEM	100
+
+#define DEFAULT_MEMORY_TIER	MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3

I understand the names are somewhat arbitrary, and the tier ID space can be expanded down the line by bumping MAX_MEMORY_TIERS.

But starting out with a packed ID space can get quite awkward for users when new tiers - especially intermediate tiers - show up in existing configurations. I mentioned in the other email that DRAM != DRAM, so new tiers seem inevitable already.

Quoted 6 lines

It could make sense to start with a bigger address space and spread out the list of kernel default tiers a bit within it:

MEMORY_TIER_GPU 0
MEMORY_TIER_DRAM 10
MEMORY_TIER_PMEM 20

Forgive me if I'm asking a question that has been answered. I went back to earlier threads and couldn't work it out - maybe there were some off-list discussions? Anyway...

Why is there a distinction between tier ID and rank? I undestand that rank was added because tier IDs were too few. But if rank determines ordering, what is the use of a separate tier ID? IOW, why not make the tier ID space wider and have the kernel pick a few spread out defaults based on known hardware, with plenty of headroom to be future proof.

   $ ls tiers
   100				# DEFAULT_TIER
   $ cat tiers/100/nodelist
   0-1				# conventional numa nodes

   <pmem is onlined>

   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1	# conventional numa
   tiers/200/nodelist:2		# pmem

   $ grep . nodes/*/tier
   nodes/0/tier:100
   nodes/1/tier:100
   nodes/2/tier:200

   <unknown device is online as node 3, defaults to 100>

   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1,3
   tiers/200/nodelist:2

   $ echo 300 >nodes/3/tier
   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1
   tiers/200/nodelist:2
   tiers/300/nodelist:3

   $ echo 200 >nodes/3/tier
   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1	
   tiers/200/nodelist:2-3

etc.

Quoted 80 lines · Johannes Weiner

Hello,

Quoted 27 lines · Johannes Weiner

Quoted 7 lines · Aneesh Kumar K.V

include/linux/memory-tiers.h new file +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

include/linux/memory-tiers.h line 16

Quoted 10 lines

include/linux/memory-tiers.h new file +10 -0

+#define MEMORY_TIER_HBM_GPU	0
+#define MEMORY_TIER_DRAM	1
+#define MEMORY_TIER_PMEM	2
+
+#define MEMORY_RANK_HBM_GPU	300
+#define MEMORY_RANK_DRAM	200
+#define MEMORY_RANK_PMEM	100
+
+#define DEFAULT_MEMORY_TIER	MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3

I understand the names are somewhat arbitrary, and the tier ID space can be expanded down the line by bumping MAX_MEMORY_TIERS.

But starting out with a packed ID space can get quite awkward for users when new tiers - especially intermediate tiers - show up in existing configurations. I mentioned in the other email that DRAM != DRAM, so new tiers seem inevitable already.

Quoted 6 lines

It could make sense to start with a bigger address space and spread out the list of kernel default tiers a bit within it:

MEMORY_TIER_GPU 0
MEMORY_TIER_DRAM 10
MEMORY_TIER_PMEM 20

Forgive me if I'm asking a question that has been answered. I went back to earlier threads and couldn't work it out - maybe there were some off-list discussions? Anyway...

Why is there a distinction between tier ID and rank? I undestand that rank was added because tier IDs were too few. But if rank determines ordering, what is the use of a separate tier ID? IOW, why not make the tier ID space wider and have the kernel pick a few spread out defaults based on known hardware, with plenty of headroom to be future proof.

   $ ls tiers
   100				# DEFAULT_TIER
   $ cat tiers/100/nodelist
   0-1				# conventional numa nodes

   <pmem is onlined>

   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1	# conventional numa
   tiers/200/nodelist:2		# pmem

   $ grep . nodes/*/tier
   nodes/0/tier:100
   nodes/1/tier:100
   nodes/2/tier:200

   <unknown device is online as node 3, defaults to 100>

   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1,3
   tiers/200/nodelist:2

   $ echo 300 >nodes/3/tier
   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1
   tiers/200/nodelist:2
   tiers/300/nodelist:3

   $ echo 200 >nodes/3/tier
   $ grep . tiers/*/nodelist
   tiers/100/nodelist:0-1	
   tiers/200/nodelist:2-3

Quoted 1 line

etc.

tier ID is also used as device id memtier.dev.id. It was discussed that we would need the ability to change the rank value of a memory tier. If we make rank value same as tier ID or tier device id, we will not be able to support that.

Quoted 81 lines · Aneesh Kumar K V

Quoted 80 lines · Johannes Weiner

Hello,

Quoted 27 lines · Johannes Weiner

Quoted 7 lines · Aneesh Kumar K.V

include/linux/memory-tiers.h new file +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

include/linux/memory-tiers.h line 16

Quoted 10 lines

include/linux/memory-tiers.h new file +10 -0

+#define MEMORY_TIER_HBM_GPU	0
+#define MEMORY_TIER_DRAM	1
+#define MEMORY_TIER_PMEM	2
+
+#define MEMORY_RANK_HBM_GPU	300
+#define MEMORY_RANK_DRAM	200
+#define MEMORY_RANK_PMEM	100
+
+#define DEFAULT_MEMORY_TIER	MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3

I understand the names are somewhat arbitrary, and the tier ID space can be expanded down the line by bumping MAX_MEMORY_TIERS.

But starting out with a packed ID space can get quite awkward for users when new tiers - especially intermediate tiers - show up in existing configurations. I mentioned in the other email that DRAM != DRAM, so new tiers seem inevitable already.

Quoted 6 lines

It could make sense to start with a bigger address space and spread out the list of kernel default tiers a bit within it:

MEMORY_TIER_GPU 0
MEMORY_TIER_DRAM 10
MEMORY_TIER_PMEM 20

Forgive me if I'm asking a question that has been answered. I went back to earlier threads and couldn't work it out - maybe there were some off-list discussions? Anyway...

Why is there a distinction between tier ID and rank? I undestand that rank was added because tier IDs were too few. But if rank determines ordering, what is the use of a separate tier ID? IOW, why not make the tier ID space wider and have the kernel pick a few spread out defaults based on known hardware, with plenty of headroom to be future proof.

    $ ls tiers
    100				# DEFAULT_TIER
    $ cat tiers/100/nodelist
    0-1				# conventional numa nodes

    <pmem is onlined>

    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1	# conventional numa
    tiers/200/nodelist:2		# pmem

    $ grep . nodes/*/tier
    nodes/0/tier:100
    nodes/1/tier:100
    nodes/2/tier:200

    <unknown device is online as node 3, defaults to 100>

    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1,3
    tiers/200/nodelist:2

    $ echo 300 >nodes/3/tier
    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1
    tiers/200/nodelist:2
    tiers/300/nodelist:3

    $ echo 200 >nodes/3/tier
    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1	
    tiers/200/nodelist:2-3

Quoted 6 lines

Quoted 1 line

etc.

tier ID is also used as device id memtier.dev.id. It was discussed that we would need the ability to change the rank value of a memory tier. If we make rank value same as tier ID or tier device id, we will not be able to support that.

Is the idea that you could change the rank of a collection of nodes in one go? Rather than moving the nodes one by one into a new tier?

[ Sorry, I wasn't able to find this discussion. AFAICS the first
patches in RFC4 already had the struct device { .id = tier } logic. Could you point me to it? In general it would be really helpful to maintain summarized rationales for such decisions in the coverletter to make sure things don't get lost over many, many threads, conferences, and video calls. ]

Quoted 92 lines · Johannes Weiner

Quoted 81 lines · Aneesh Kumar K V

Quoted 80 lines · Johannes Weiner

Hello,

Quoted 27 lines · Johannes Weiner

Quoted 7 lines · Aneesh Kumar K.V

include/linux/memory-tiers.h new file +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

include/linux/memory-tiers.h line 16

Quoted 10 lines

include/linux/memory-tiers.h new file +10 -0

+#define MEMORY_TIER_HBM_GPU	0
+#define MEMORY_TIER_DRAM	1
+#define MEMORY_TIER_PMEM	2
+
+#define MEMORY_RANK_HBM_GPU	300
+#define MEMORY_RANK_DRAM	200
+#define MEMORY_RANK_PMEM	100
+
+#define DEFAULT_MEMORY_TIER	MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3

I understand the names are somewhat arbitrary, and the tier ID space can be expanded down the line by bumping MAX_MEMORY_TIERS.

But starting out with a packed ID space can get quite awkward for users when new tiers - especially intermediate tiers - show up in existing configurations. I mentioned in the other email that DRAM != DRAM, so new tiers seem inevitable already.

Quoted 6 lines

It could make sense to start with a bigger address space and spread out the list of kernel default tiers a bit within it:

MEMORY_TIER_GPU 0
MEMORY_TIER_DRAM 10
MEMORY_TIER_PMEM 20

Forgive me if I'm asking a question that has been answered. I went back to earlier threads and couldn't work it out - maybe there were some off-list discussions? Anyway...

Why is there a distinction between tier ID and rank? I undestand that rank was added because tier IDs were too few. But if rank determines ordering, what is the use of a separate tier ID? IOW, why not make the tier ID space wider and have the kernel pick a few spread out defaults based on known hardware, with plenty of headroom to be future proof.

    $ ls tiers
    100				# DEFAULT_TIER
    $ cat tiers/100/nodelist
    0-1				# conventional numa nodes

    <pmem is onlined>

    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1	# conventional numa
    tiers/200/nodelist:2		# pmem

    $ grep . nodes/*/tier
    nodes/0/tier:100
    nodes/1/tier:100
    nodes/2/tier:200

    <unknown device is online as node 3, defaults to 100>

    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1,3
    tiers/200/nodelist:2

    $ echo 300 >nodes/3/tier
    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1
    tiers/200/nodelist:2
    tiers/300/nodelist:3

    $ echo 200 >nodes/3/tier
    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1	
    tiers/200/nodelist:2-3

Quoted 6 lines

Quoted 1 line

etc.

tier ID is also used as device id memtier.dev.id. It was discussed that we would need the ability to change the rank value of a memory tier. If we make rank value same as tier ID or tier device id, we will not be able to support that.

Is the idea that you could change the rank of a collection of nodes in one go? Rather than moving the nodes one by one into a new tier?

Quoted 6 lines

[ Sorry, I wasn't able to find this discussion. AFAICS the first
patches in RFC4 already had the struct device { .id = tier } logic. Could you point me to it? In general it would be really helpful to maintain summarized rationales for such decisions in the coverletter to make sure things don't get lost over many, many threads, conferences, and video calls. ]

Most of the discussion happened not int he patch review email threads.

RFC: Memory Tiering Kernel Interfaces (v2)
https://lore.kernel.org/linux-mm/CAAPL-u_diGYEb7+WsgqNBLRix-nRCk2SsDj6p9r8j5JZwOABZQ@mail.gmail.com

RFC: Memory Tiering Kernel Interfaces (v4)
https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

Quoted 100 lines · Aneesh Kumar K V

Quoted 92 lines · Johannes Weiner

Quoted 81 lines · Aneesh Kumar K V

Quoted 80 lines · Johannes Weiner

Hello,

Quoted 27 lines · Johannes Weiner

Quoted 7 lines · Aneesh Kumar K.V

(quoted diff) +6 -0

@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+

Review line 16

Quoted 10 lines

(quoted diff) +10 -0

+#define MEMORY_TIER_HBM_GPU	0
+#define MEMORY_TIER_DRAM	1
+#define MEMORY_TIER_PMEM	2
+
+#define MEMORY_RANK_HBM_GPU	300
+#define MEMORY_RANK_DRAM	200
+#define MEMORY_RANK_PMEM	100
+
+#define DEFAULT_MEMORY_TIER	MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS  3  

I understand the names are somewhat arbitrary, and the tier ID space can be expanded down the line by bumping MAX_MEMORY_TIERS.

But starting out with a packed ID space can get quite awkward for users when new tiers - especially intermediate tiers - show up in existing configurations. I mentioned in the other email that DRAM != DRAM, so new tiers seem inevitable already.

Quoted 6 lines

It could make sense to start with a bigger address space and spread out the list of kernel default tiers a bit within it:

MEMORY_TIER_GPU 0
MEMORY_TIER_DRAM 10
MEMORY_TIER_PMEM 20

Forgive me if I'm asking a question that has been answered. I went back to earlier threads and couldn't work it out - maybe there were some off-list discussions? Anyway...

Why is there a distinction between tier ID and rank? I undestand that rank was added because tier IDs were too few. But if rank determines ordering, what is the use of a separate tier ID? IOW, why not make the tier ID space wider and have the kernel pick a few spread out defaults based on known hardware, with plenty of headroom to be future proof.

    $ ls tiers
    100				# DEFAULT_TIER
    $ cat tiers/100/nodelist
    0-1				# conventional numa nodes

    <pmem is onlined>

    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1	# conventional numa
    tiers/200/nodelist:2		# pmem

    $ grep . nodes/*/tier
    nodes/0/tier:100
    nodes/1/tier:100
    nodes/2/tier:200

    <unknown device is online as node 3, defaults to 100>

    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1,3
    tiers/200/nodelist:2

    $ echo 300 >nodes/3/tier
    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1
    tiers/200/nodelist:2
    tiers/300/nodelist:3

    $ echo 200 >nodes/3/tier
    $ grep . tiers/*/nodelist
    tiers/100/nodelist:0-1	
    tiers/200/nodelist:2-3

Quoted 6 lines

Quoted 1 line

etc.

tier ID is also used as device id memtier.dev.id. It was discussed that we would need the ability to change the rank value of a memory tier. If we make rank value same as tier ID or tier device id, we will not be able to support that.

Is the idea that you could change the rank of a collection of nodes in one go? Rather than moving the nodes one by one into a new tier?

Quoted 6 lines

[ Sorry, I wasn't able to find this discussion. AFAICS the first
patches in RFC4 already had the struct device { .id = tier } logic. Could you point me to it? In general it would be really helpful to maintain summarized rationales for such decisions in the coverletter to make sure things don't get lost over many, many threads, conferences, and video calls. ]

Quoted 7 lines

Most of the discussion happened not int he patch review email threads.

RFC: Memory Tiering Kernel Interfaces (v2)
https://lore.kernel.org/linux-mm/CAAPL-u_diGYEb7+WsgqNBLRix-nRCk2SsDj6p9r8j5JZwOABZQ@mail.gmail.com

RFC: Memory Tiering Kernel Interfaces (v4)
https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com

I read the RFCs, the discussions and your code. It's still not clear why the tier/device ID and the rank need to be two separate, user-visible things. There is only one tier of a given rank, why can't the rank be the unique device id? dev->id = 100. One number. Or use a unique device id allocator if large numbers are causing problems internally. But I don't see an explanation why they need to be two different things, let alone two different things in the user ABI.

I think discussion hinged on it making sense to be able to change rank of a tier rather than create a new tier and move things one by one. Example was wanting to change the rank of a tier that was created either by core code or a subsystem.

E.g. If GPU driver creates a tier, assumption is all similar GPUs will default to the same tier (if hot plugged later for example) as the driver subsystem will keep a reference to the created tier. Hence if user wants to change the order of that relative to other tiers, the option of creating a new tier and moving the devices would then require us to have infrastructure to tell the GPU driver to now use the new tier for additional devices.

Quoted 5 lines · Jonathan Cameron

I think discussion hinged on it making sense to be able to change rank of a tier rather than create a new tier and move things one by one. Example was wanting to change the rank of a tier that was created either by core code or a subsystem.

Quoted 7 lines

E.g. If GPU driver creates a tier, assumption is all similar GPUs will default to the same tier (if hot plugged later for example) as the driver subsystem will keep a reference to the created tier. Hence if user wants to change the order of that relative to other tiers, the option of creating a new tier and moving the devices would then require us to have infrastructure to tell the GPU driver to now use the new tier for additional devices.

That's an interesting point, thanks for explaining.

But that could still happen when two drivers report the same tier and one of them is wrong, right? You'd still need to separate out by hand to adjust rank, as well as handle hotplug events. Driver colllisions are probable with coarse categories like gpu, dram, pmem.

Would it make more sense to have the platform/devicetree/driver provide more fine-grained distance values similar to NUMA distances, and have a driver-scope tunable to override/correct? And then have the distance value function as the unique tier ID and rank in one.

That would allow device class reassignments, too, and it would work with driver collisions where simple "tier stickiness" would not. (Although collisions would be less likely to begin with given a broader range of possible distance values.)

Going further, it could be useful to separate the business of hardware properties (and configuring quirks) from the business of configuring MM policies that should be applied to the resulting tier hierarchy. They're somewhat orthogonal tuning tasks, and one of them might become obsolete before the other (if the quality of distance values provided by drivers improves before the quality of MM heuristics ;). Separating them might help clarify the interface for both designers and users.

E.g. a memdev class scope with a driver-wide distance value, and a memdev scope for per-device values that default to "inherit driver value". The memtier subtree would then have an r/o structure, but allow tuning per-tier interleaving ratio[1], demotion rules etc.

[1] https://lore.kernel.org/linux-mm/[email protected]/#t

Quoted 5 lines · Jonathan Cameron

I think discussion hinged on it making sense to be able to change rank of a tier rather than create a new tier and move things one by one. Example was wanting to change the rank of a tier that was created either by core code or a subsystem.

Quoted 7 lines

E.g. If GPU driver creates a tier, assumption is all similar GPUs will default to the same tier (if hot plugged later for example) as the driver subsystem will keep a reference to the created tier. Hence if user wants to change the order of that relative to other tiers, the option of creating a new tier and moving the devices would then require us to have infrastructure to tell the GPU driver to now use the new tier for additional devices.

That's an interesting point, thanks for explaining.

But that could still happen when two drivers report the same tier and one of them is wrong, right? You'd still need to separate out by hand to adjust rank, as well as handle hotplug events. Driver colllisions are probable with coarse categories like gpu, dram, pmem.

Would it make more sense to have the platform/devicetree/driver provide more fine-grained distance values similar to NUMA distances, and have a driver-scope tunable to override/correct? And then have the distance value function as the unique tier ID and rank in one.

That would allow device class reassignments, too, and it would work with driver collisions where simple "tier stickiness" would not. (Although collisions would be less likely to begin with given a broader range of possible distance values.)

Going further, it could be useful to separate the business of hardware properties (and configuring quirks) from the business of configuring MM policies that should be applied to the resulting tier hierarchy. They're somewhat orthogonal tuning tasks, and one of them might become obsolete before the other (if the quality of distance values provided by drivers improves before the quality of MM heuristics ;). Separating them might help clarify the interface for both designers and users.

E.g. a memdev class scope with a driver-wide distance value, and a memdev scope for per-device values that default to "inherit driver value". The memtier subtree would then have an r/o structure, but allow tuning per-tier interleaving ratio[1], demotion rules etc.

[1] https://lore.kernel.org/linux-mm/[email protected]/#t

Quoted 5 lines · On Thu, 9 Jun 2022 16:41:04 -0400 Johannes Weiner <[email protected]> wrote:

On Thu, Jun 09, 2022 at 03:22:43PM +0100, Jonathan Cameron wrote:
Would it make more sense to have the platform/devicetree/driver provide more fine-grained distance values similar to NUMA distances, and have a driver-scope tunable to override/correct? And then have the distance value function as the unique tier ID and rank in one.

Absolutely a good thing to provide that information, but it's black magic. There are too many contradicting metrics (latency vs bandwidth etc) even not including a more complex system model like Jerome Glisse proposed a few years back. https://lore.kernel.org/all/[email protected]/
CXL 2.0 got this more right than anything else I've seen as provides discoverable topology along with details like latency to cross between particular switch ports. Actually using that data (other than by throwing it to userspace controls for HPC apps etc) is going to take some figuring out. Even the question of what + how we expose this info to userspace is non obvious.

Quoted 8 lines

Going further, it could be useful to separate the business of hardware properties (and configuring quirks) from the business of configuring MM policies that should be applied to the resulting tier hierarchy. They're somewhat orthogonal tuning tasks, and one of them might become obsolete before the other (if the quality of distance values provided by drivers improves before the quality of MM heuristics ;). Separating them might help clarify the interface for both designers and users.

Quoted 4 lines

E.g. a memdev class scope with a driver-wide distance value, and a memdev scope for per-device values that default to "inherit driver value". The memtier subtree would then have an r/o structure, but allow tuning per-tier interleaving ratio[1], demotion rules etc.

Ok that makes sense. I'm not sure if that ends up as an implementation detail, or effects the userspace interface of this particular element.

I'm not sure completely read only is flexible enough (though mostly RO is fine) as we keep sketching out cases where any attempt to do things automatically does the wrong thing and where we need to add an extra tier to get everything to work. Short of having a lot of tiers I'm not sure how we could have the default work well. Maybe a lot of "tiers" is fine though perhaps we need to rename them if going this way and then they don't really work as current concept of tier.

Imagine a system with subtle difference between different memories such as 10% latency increase for same bandwidth. To get an advantage from demoting to such a tier will require really stable usage and long run times. Whilst you could design a demotion scheme that takes that into account, I think we are a long way from that today.

Quoted 1 line · Jonathan Cameron

Quoted 8 lines

I'm not sure completely read only is flexible enough (though mostly RO is fine) as we keep sketching out cases where any attempt to do things automatically does the wrong thing and where we need to add an extra tier to get everything to work. Short of having a lot of tiers I'm not sure how we could have the default work well. Maybe a lot of "tiers" is fine though perhaps we need to rename them if going this way and then they don't really work as current concept of tier.

Quoted 5 lines

Imagine a system with subtle difference between different memories such as 10% latency increase for same bandwidth. To get an advantage from demoting to such a tier will require really stable usage and long run times. Whilst you could design a demotion scheme that takes that into account, I think we are a long way from that today.

Good point: there can be a clear hardware difference, but it's a policy choice whether the MM should treat them as one or two tiers.

What do you think of a per-driver/per-device (overridable) distance number, combined with a configurable distance cutoff for what constitutes separate tiers. E.g. cutoff=20 means two devices with distances of 10 and 20 respectively would be in the same tier, devices with 10 and 100 would be in separate ones. The kernel then generates and populates the tiers based on distances and grouping cutoff, and populates the memtier directory tree and nodemasks in sysfs.

It could be simple tier0, tier1, tier2 numbering again, but the numbers now would mean something to the user. A rank tunable is no longer necessary.

I think even the nodemasks in the memtier tree could be read-only then, since corrections should only be necessary when either the device distance is wrong or the tier grouping cutoff.

Can you think of scenarios where that scheme would fall apart?

Quoted 18 lines · Johannes Weiner

Quoted 8 lines · Jonathan Cameron

I'm not sure completely read only is flexible enough (though mostly RO is fine) as we keep sketching out cases where any attempt to do things automatically does the wrong thing and where we need to add an extra tier to get everything to work. Short of having a lot of tiers I'm not sure how we could have the default work well. Maybe a lot of "tiers" is fine though perhaps we need to rename them if going this way and then they don't really work as current concept of tier.

Quoted 5 lines

Imagine a system with subtle difference between different memories such as 10% latency increase for same bandwidth. To get an advantage from demoting to such a tier will require really stable usage and long run times. Whilst you could design a demotion scheme that takes that into account, I think we are a long way from that today.

Good point: there can be a clear hardware difference, but it's a policy choice whether the MM should treat them as one or two tiers.

Quoted 8 lines

What do you think of a per-driver/per-device (overridable) distance number, combined with a configurable distance cutoff for what constitutes separate tiers. E.g. cutoff=20 means two devices with distances of 10 and 20 respectively would be in the same tier, devices with 10 and 100 would be in separate ones. The kernel then generates and populates the tiers based on distances and grouping cutoff, and populates the memtier directory tree and nodemasks in sysfs.

Right now core/generic code doesn't get involved in building tiers. It just defines three tiers where drivers could place the respective devices they manage. The above suggestion would imply we are moving quite a lot of policy decision logic into the generic code?.

At some point, we will have to depend on more attributes other than distance(may be HMAT?) and each driver should have the flexibility to place the device it is managing in a specific tier? By then we may decide to support more than 3 static tiers which the core kernel currently does.

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

Quoted 28 lines · Aneesh Kumar K V

Quoted 18 lines · Johannes Weiner

Quoted 8 lines · Jonathan Cameron

I'm not sure completely read only is flexible enough (though mostly RO is fine) as we keep sketching out cases where any attempt to do things automatically does the wrong thing and where we need to add an extra tier to get everything to work. Short of having a lot of tiers I'm not sure how we could have the default work well. Maybe a lot of "tiers" is fine though perhaps we need to rename them if going this way and then they don't really work as current concept of tier.

Quoted 5 lines

Imagine a system with subtle difference between different memories such as 10% latency increase for same bandwidth. To get an advantage from demoting to such a tier will require really stable usage and long run times. Whilst you could design a demotion scheme that takes that into account, I think we are a long way from that today.

Good point: there can be a clear hardware difference, but it's a policy choice whether the MM should treat them as one or two tiers.

Quoted 8 lines

What do you think of a per-driver/per-device (overridable) distance number, combined with a configurable distance cutoff for what constitutes separate tiers. E.g. cutoff=20 means two devices with distances of 10 and 20 respectively would be in the same tier, devices with 10 and 100 would be in separate ones. The kernel then generates and populates the tiers based on distances and grouping cutoff, and populates the memtier directory tree and nodemasks in sysfs.

Quoted 4 lines

Right now core/generic code doesn't get involved in building tiers. It just defines three tiers where drivers could place the respective devices they manage. The above suggestion would imply we are moving quite a lot of policy decision logic into the generic code?.

No. The driver still chooses its own number, just from a wider range. The only policy in generic code is the distance cutoff for which devices are grouped into tiers together.

Quoted 4 lines

At some point, we will have to depend on more attributes other than distance(may be HMAT?) and each driver should have the flexibility to place the device it is managing in a specific tier? By then we may decide to support more than 3 static tiers which the core kernel currently does.

If we start with a larger possible range of "distance" values right away, we can still let the drivers ballpark into 3 tiers for now (100, 200, 300). But it will be easier to take additional metrics into account later and fine tune accordingly (120, 260, 90 etc.) without having to update all the other drivers as well.

Quoted 8 lines

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

I had touched on this in the other email.

This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface.

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers.

With these parameters alone, you can generate an ordered list of tiers and their devices. The tier numbers make sense, and no rank is needed.

Do you still need the ability to move nodes by writing nodemasks? I don't think so. Assuming you would never want to have an actually slower device in a higher tier than a faster device, the only time you'd want to move a device is when the device's distance value is wrong. So you override that (until you update to a fixed kernel).

Quoted 28 lines · Aneesh Kumar K V

Quoted 18 lines · Johannes Weiner

Quoted 8 lines · Jonathan Cameron

I'm not sure completely read only is flexible enough (though mostly RO is fine) as we keep sketching out cases where any attempt to do things automatically does the wrong thing and where we need to add an extra tier to get everything to work. Short of having a lot of tiers I'm not sure how we could have the default work well. Maybe a lot of "tiers" is fine though perhaps we need to rename them if going this way and then they don't really work as current concept of tier.

Quoted 5 lines

Imagine a system with subtle difference between different memories such as 10% latency increase for same bandwidth. To get an advantage from demoting to such a tier will require really stable usage and long run times. Whilst you could design a demotion scheme that takes that into account, I think we are a long way from that today.

Good point: there can be a clear hardware difference, but it's a policy choice whether the MM should treat them as one or two tiers.

Quoted 8 lines

What do you think of a per-driver/per-device (overridable) distance number, combined with a configurable distance cutoff for what constitutes separate tiers. E.g. cutoff=20 means two devices with distances of 10 and 20 respectively would be in the same tier, devices with 10 and 100 would be in separate ones. The kernel then generates and populates the tiers based on distances and grouping cutoff, and populates the memtier directory tree and nodemasks in sysfs.

Quoted 4 lines

Right now core/generic code doesn't get involved in building tiers. It just defines three tiers where drivers could place the respective devices they manage. The above suggestion would imply we are moving quite a lot of policy decision logic into the generic code?.

No. The driver still chooses its own number, just from a wider range. The only policy in generic code is the distance cutoff for which devices are grouped into tiers together.

Quoted 4 lines

At some point, we will have to depend on more attributes other than distance(may be HMAT?) and each driver should have the flexibility to place the device it is managing in a specific tier? By then we may decide to support more than 3 static tiers which the core kernel currently does.

If we start with a larger possible range of "distance" values right away, we can still let the drivers ballpark into 3 tiers for now (100, 200, 300). But it will be easier to take additional metrics into account later and fine tune accordingly (120, 260, 90 etc.) without having to update all the other drivers as well.

Quoted 8 lines

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

I had touched on this in the other email.

This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface.

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers.

With these parameters alone, you can generate an ordered list of tiers and their devices. The tier numbers make sense, and no rank is needed.

Do you still need the ability to move nodes by writing nodemasks? I don't think so. Assuming you would never want to have an actually slower device in a higher tier than a faster device, the only time you'd want to move a device is when the device's distance value is wrong. So you override that (until you update to a fixed kernel).

Quoted 10 lines · Johannes Weiner

Quoted 8 lines · Aneesh Kumar K V

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

Quoted 8 lines

I had touched on this in the other email.

This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface.

Considering the kernel has a static map for these tiers, how can two drivers end up using the same tier? If a new driver is going to manage a memory device that is of different characteristics than the one managed by dax/kmem, we will end up adding

#define MEMORY_TIER_NEW_DEVICE 4

The new driver will never use MEMORY_TIER_PMEM

What can happen is two devices that are managed by DAX/kmem that should be in two memory tiers get assigned the same memory tier because the dax/kmem driver added both the device to the same memory tier.

In the future we would avoid that by using more device properties like HMAT to create additional memory tiers with different rank values. ie, we would do in the dax/kmem create_tier_from_rank() .

Quoted 4 lines

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

I understand that the driver override will be done via module parameters. How will we implement device override? For example in case of dax/kmem driver the device override will be per dax device? What interface will we use to set the override?

IIUC in the above proposal the dax/kmem will do

node_create_and_set_memory_tier(numa_node, get_device_tier_index(dev_dax));

get_device_tier_index(struct dev_dax *dev) {
return dax_kmem_tier_index; // module parameter
}

Are you suggesting to add a dev_dax property to override the tier defaults?

Quoted 4 lines

This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers.

Can you elaborate more on how distance value will be used? The device/device NUMA node can have different distance value from other NUMA nodes. How do we group them?
for ex: earlier discussion did outline three different topologies. Can you ellaborate how we would end up grouping them using distance?

For ex: in the topology below node 2 is at distance 30 from Node0 and 40 from Nodes so how will we classify node 2?

Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes.

		  20
  Node 0 (DRAM)  ----  Node 1 (DRAM)
       |        \   /       |
       | 30    40 X 40      | 30
       |        /   \       |
  Node 2 (PMEM)  ----  Node 3 (PMEM)
		  40

node distances:
node 0 1 2 3

   0  10   20   30   40
   1  20   10   40   30
   2  30   40   10   40
   3  40   30   40   10

Quoted 7 lines

What can happen is two devices that are managed by DAX/kmem that should be in two memory tiers get assigned the same memory tier because the dax/kmem driver added both the device to the same memory tier.

In the future we would avoid that by using more device properties like HMAT to create additional memory tiers with different rank values. ie, we would do in the dax/kmem create_tier_from_rank() .

Yes, that's the type of collision I mean. Two GPUs, two CXL-attached DRAMs of different speeds etc.

I also like Huang's idea of using latency characteristics instead of abstract distances. Though I'm not quite sure how feasible this is in the short term, and share some concerns that Jonathan raised. But I think a wider possible range to begin with makes sense in any case.

Quoted 4 lines

Quoted 4 lines

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

Quoted 20 lines

Can you elaborate more on how distance value will be used? The device/device NUMA node can have different distance value from other NUMA nodes. How do we group them?
for ex: earlier discussion did outline three different topologies. Can you ellaborate how we would end up grouping them using distance?

For ex: in the topology below node 2 is at distance 30 from Node0 and 40 from Nodes so how will we classify node 2?

Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes.

		  20
  Node 0 (DRAM)  ----  Node 1 (DRAM)
       |        \   /       |
       | 30    40 X 40      | 30
       |        /   \       |
  Node 2 (PMEM)  ----  Node 3 (PMEM)
		  40

Quoted 6 lines

node distances:
node 0 1 2 3

   0  10   20   30   40
   1  20   10   40   30
   2  30   40   10   40
   3  40   30   40   10

I'm fairly confused by this example. Do all nodes have CPUs? Isn't this just classic NUMA, where optimizing for locality makes the most sense, rather than tiering?

Forget the interface for a second, I have no idea how tiering on such a system would work. One CPU's lower tier can be another CPU's toptier. There is no lowest rung from which to actually *reclaim* pages. Would the CPUs just demote in circles?

And the coldest pages on one socket would get demoted into another socket and displace what that socket considers hot local memory?

I feel like I missing something.

When we're talking about tiered memory, I'm thinking about CPUs utilizing more than one memory node. If those other nodes have CPUs, you can't reliably establish a singular tier order anymore and it becomes classic NUMA, no?

Quoted 29 lines · Aneesh Kumar K V

Quoted 10 lines · Johannes Weiner

Quoted 8 lines · Aneesh Kumar K V

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

Quoted 8 lines

I had touched on this in the other email.

This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface.

Considering the kernel has a static map for these tiers, how can two drivers end up using the same tier? If a new driver is going to manage a memory device that is of different characteristics than the one managed by dax/kmem, we will end up adding

#define MEMORY_TIER_NEW_DEVICE 4

The new driver will never use MEMORY_TIER_PMEM

Quoted 7 lines

What can happen is two devices that are managed by DAX/kmem that should be in two memory tiers get assigned the same memory tier because the dax/kmem driver added both the device to the same memory tier.

In the future we would avoid that by using more device properties like HMAT to create additional memory tiers with different rank values. ie, we would do in the dax/kmem create_tier_from_rank() .

Yes, that's the type of collision I mean. Two GPUs, two CXL-attached DRAMs of different speeds etc.

I also like Huang's idea of using latency characteristics instead of abstract distances. Though I'm not quite sure how feasible this is in the short term, and share some concerns that Jonathan raised. But I think a wider possible range to begin with makes sense in any case.

Quoted 11 lines

Quoted 4 lines

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

I understand that the driver override will be done via module parameters. How will we implement device override? For example in case of dax/kmem driver the device override will be per dax device? What interface will we use to set the override?

IIUC in the above proposal the dax/kmem will do

Quoted 8 lines

node_create_and_set_memory_tier(numa_node, get_device_tier_index(dev_dax));

get_device_tier_index(struct dev_dax *dev) {
    return dax_kmem_tier_index; // module parameter }

Are you suggesting to add a dev_dax property to override the tier defaults?

I was thinking a new struct memdevice and struct memtype(?). Every driver implementing memory devices like this sets those up and registers them with generic code and preset parameters. The generic code creates sysfs directories and allows overriding the parameters.

struct memdevice {
	struct device dev;
	unsigned long distance;
	struct list_head siblings;
	/* nid? ... */
};

struct memtype {
	struct device_type type;
	unsigned long default_distance;
	struct list_head devices;
};

That forms the (tweakable) tree describing physical properties.

From that, the kernel then generates the ordered list of tiers.

Quoted 24 lines

Quoted 4 lines

This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers.

Can you elaborate more on how distance value will be used? The device/device NUMA node can have different distance value from other NUMA nodes. How do we group them?
for ex: earlier discussion did outline three different topologies. Can you ellaborate how we would end up grouping them using distance?

For ex: in the topology below node 2 is at distance 30 from Node0 and 40 from Nodes so how will we classify node 2?

Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes.

		  20
  Node 0 (DRAM)  ----  Node 1 (DRAM)
       |        \   /       |
       | 30    40 X 40      | 30
       |        /   \       |
  Node 2 (PMEM)  ----  Node 3 (PMEM)
		  40

Quoted 6 lines

node distances:
node 0 1 2 3

   0  10   20   30   40
   1  20   10   40   30
   2  30   40   10   40
   3  40   30   40   10

I'm fairly confused by this example. Do all nodes have CPUs? Isn't this just classic NUMA, where optimizing for locality makes the most sense, rather than tiering?

Forget the interface for a second, I have no idea how tiering on such a system would work. One CPU's lower tier can be another CPU's toptier. There is no lowest rung from which to actually *reclaim* pages. Would the CPUs just demote in circles?

And the coldest pages on one socket would get demoted into another socket and displace what that socket considers hot local memory?

I feel like I missing something.

When we're talking about tiered memory, I'm thinking about CPUs utilizing more than one memory node. If those other nodes have CPUs, you can't reliably establish a singular tier order anymore and it becomes classic NUMA, no?

Quoted 78 lines · Johannes Weiner

Quoted 29 lines · Aneesh Kumar K V

Quoted 10 lines · Johannes Weiner

Quoted 8 lines · Aneesh Kumar K V

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

Quoted 8 lines

I had touched on this in the other email.

This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface.

Considering the kernel has a static map for these tiers, how can two drivers end up using the same tier? If a new driver is going to manage a memory device that is of different characteristics than the one managed by dax/kmem, we will end up adding

#define MEMORY_TIER_NEW_DEVICE 4

The new driver will never use MEMORY_TIER_PMEM

Quoted 7 lines

What can happen is two devices that are managed by DAX/kmem that should be in two memory tiers get assigned the same memory tier because the dax/kmem driver added both the device to the same memory tier.

In the future we would avoid that by using more device properties like HMAT to create additional memory tiers with different rank values. ie, we would do in the dax/kmem create_tier_from_rank() .

Yes, that's the type of collision I mean. Two GPUs, two CXL-attached DRAMs of different speeds etc.

I also like Huang's idea of using latency characteristics instead of abstract distances. Though I'm not quite sure how feasible this is in the short term, and share some concerns that Jonathan raised. But I think a wider possible range to begin with makes sense in any case.

Quoted 11 lines

Quoted 4 lines

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

I understand that the driver override will be done via module parameters. How will we implement device override? For example in case of dax/kmem driver the device override will be per dax device? What interface will we use to set the override?

IIUC in the above proposal the dax/kmem will do

Quoted 8 lines

node_create_and_set_memory_tier(numa_node, get_device_tier_index(dev_dax));

get_device_tier_index(struct dev_dax *dev) {
return dax_kmem_tier_index; // module parameter
}

Are you suggesting to add a dev_dax property to override the tier defaults?

I was thinking a new struct memdevice and struct memtype(?). Every driver implementing memory devices like this sets those up and registers them with generic code and preset parameters. The generic code creates sysfs directories and allows overriding the parameters.

struct memdevice {
      struct device dev;
      unsigned long distance;
      struct list_head siblings;
      /* nid? ... */
};

Quoted 7 lines

struct memtype {
      struct device_type type;
      unsigned long default_distance;
      struct list_head devices;
};

That forms the (tweakable) tree describing physical properties.

In general, I think memtype is a good idea. I have suggested something similar before. It can describe the characters of a specific type of memory (same memory media with different interface (e.g., CXL, or DIMM) will be different memory types). And they can be used to provide overriding information.

As for memdevice, I think that we already have "node" to represent them in sysfs. Do we really need another one? Is it sufficient to add some links to node in the appropriate directory? For example, make memtype class device under the physical device (e.g. CXL device), and create links to node inside the memtype class device directory?

Quoted 1 line

From that, the kernel then generates the ordered list of tiers.

As Jonathan Cameron pointed, we may need the memory tier ID to be stable if possible. I know this isn't a easy task. At least we can make the default memory tier (CPU local DRAM) ID stable (for example make it always 128)? That provides an anchor for users to understand.

Best Regards,
Huang, Ying

Quoted 49 lines

Quoted 24 lines

Quoted 4 lines

This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers.

Can you elaborate more on how distance value will be used? The device/device NUMA node can have different distance value from other NUMA nodes. How do we group them?
for ex: earlier discussion did outline three different topologies. Can you ellaborate how we would end up grouping them using distance?

For ex: in the topology below node 2 is at distance 30 from Node0 and 40 from Nodes so how will we classify node 2?

Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes.

              20
  Node 0 (DRAM)  ----  Node 1 (DRAM)
       |        \   /       |
       | 30    40 X 40      | 30
       |        /   \       |
  Node 2 (PMEM)  ----  Node 3 (PMEM)
              40

Quoted 6 lines

node distances:
node 0 1 2 3

   0  10   20   30   40
   1  20   10   40   30
   2  30   40   10   40
   3  40   30   40   10

I'm fairly confused by this example. Do all nodes have CPUs? Isn't this just classic NUMA, where optimizing for locality makes the most sense, rather than tiering?

Forget the interface for a second, I have no idea how tiering on such a system would work. One CPU's lower tier can be another CPU's toptier. There is no lowest rung from which to actually *reclaim* pages. Would the CPUs just demote in circles?

And the coldest pages on one socket would get demoted into another socket and displace what that socket considers hot local memory?

I feel like I missing something.

When we're talking about tiered memory, I'm thinking about CPUs utilizing more than one memory node. If those other nodes have CPUs, you can't reliably establish a singular tier order anymore and it becomes classic NUMA, no?

Quoted 3 lines · Ying Huang

Quoted 1 line · Johannes Weiner

On Tue, Jun 14, 2022 at 01:31:37PM +0530, Aneesh Kumar K V wrote:

Quoted 4 lines

As Jonathan Cameron pointed, we may need the memory tier ID to be stable if possible. I know this isn't a easy task. At least we can make the default memory tier (CPU local DRAM) ID stable (for example make it always 128)? That provides an anchor for users to understand.

One of the motivations of introducing "rank" is to allow memory tier ID to be stable, at least for the well-defined tiers such as the default memory tier. The default memory tier can be moved around in the tier hierarchy by adjusting its rank position relative to other tiers, but its device ID can remain the same, e.g. always 1.

Quoted 4 lines · Wei Xu

Quoted 3 lines · Ying Huang

Quoted 1 line · Johannes Weiner

On Tue, Jun 14, 2022 at 01:31:37PM +0530, Aneesh Kumar K V wrote:

....

Quoted 11 lines

Quoted 4 lines

As Jonathan Cameron pointed, we may need the memory tier ID to be stable if possible. I know this isn't a easy task. At least we can make the default memory tier (CPU local DRAM) ID stable (for example make it always 128)? That provides an anchor for users to understand.

One of the motivations of introducing "rank" is to allow memory tier ID to be stable, at least for the well-defined tiers such as the default memory tier. The default memory tier can be moved around in the tier hierarchy by adjusting its rank position relative to other tiers, but its device ID can remain the same, e.g. always 1.

With /sys/devices/system/memtier/default_tier userspace will be able query the default tier details.

Quoted 84 lines · Johannes Weiner

Quoted 29 lines · Aneesh Kumar K V

Quoted 10 lines · Johannes Weiner

Quoted 8 lines · Aneesh Kumar K V

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

Quoted 8 lines

I had touched on this in the other email.

This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface.

Considering the kernel has a static map for these tiers, how can two drivers end up using the same tier? If a new driver is going to manage a memory device that is of different characteristics than the one managed by dax/kmem, we will end up adding

#define MEMORY_TIER_NEW_DEVICE 4

The new driver will never use MEMORY_TIER_PMEM

Quoted 7 lines

What can happen is two devices that are managed by DAX/kmem that should be in two memory tiers get assigned the same memory tier because the dax/kmem driver added both the device to the same memory tier.

In the future we would avoid that by using more device properties like HMAT to create additional memory tiers with different rank values. ie, we would do in the dax/kmem create_tier_from_rank() .

Yes, that's the type of collision I mean. Two GPUs, two CXL-attached DRAMs of different speeds etc.

I also like Huang's idea of using latency characteristics instead of abstract distances. Though I'm not quite sure how feasible this is in the short term, and share some concerns that Jonathan raised. But I think a wider possible range to begin with makes sense in any case.

Quoted 11 lines

Quoted 4 lines

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

I understand that the driver override will be done via module parameters. How will we implement device override? For example in case of dax/kmem driver the device override will be per dax device? What interface will we use to set the override?

IIUC in the above proposal the dax/kmem will do

Quoted 8 lines

node_create_and_set_memory_tier(numa_node, get_device_tier_index(dev_dax));

get_device_tier_index(struct dev_dax *dev) {
    return dax_kmem_tier_index; // module parameter }

Are you suggesting to add a dev_dax property to override the tier defaults?

I was thinking a new struct memdevice and struct memtype(?). Every driver implementing memory devices like this sets those up and registers them with generic code and preset parameters. The generic code creates sysfs directories and allows overriding the parameters.

struct memdevice {
	struct device dev;
	unsigned long distance;
	struct list_head siblings;
	/* nid? ... */
};

struct memtype {
	struct device_type type;
	unsigned long default_distance;
	struct list_head devices;
};

Quoted 1 line

That forms the (tweakable) tree describing physical properties.

In general, I think memtype is a good idea. I have suggested something similar before. It can describe the characters of a specific type of memory (same memory media with different interface (e.g., CXL, or DIMM) will be different memory types). And they can be used to provide overriding information.

As for memdevice, I think that we already have "node" to represent them in sysfs. Do we really need another one? Is it sufficient to add some links to node in the appropriate directory? For example, make memtype class device under the physical device (e.g. CXL device), and create links to node inside the memtype class device directory?

Quoted 1 line

From that, the kernel then generates the ordered list of tiers.

As Jonathan Cameron pointed, we may need the memory tier ID to be stable if possible. I know this isn't a easy task. At least we can make the default memory tier (CPU local DRAM) ID stable (for example make it always 128)? That provides an anchor for users to understand.

Best Regards,
Huang, Ying

Quoted 49 lines

Quoted 24 lines

Quoted 4 lines

This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers.

Can you elaborate more on how distance value will be used? The device/device NUMA node can have different distance value from other NUMA nodes. How do we group them?
for ex: earlier discussion did outline three different topologies. Can you ellaborate how we would end up grouping them using distance?

For ex: in the topology below node 2 is at distance 30 from Node0 and 40 from Nodes so how will we classify node 2?

Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes.

		  20
  Node 0 (DRAM)  ----  Node 1 (DRAM)
       |        \   /       |
       | 30    40 X 40      | 30
       |        /   \       |
  Node 2 (PMEM)  ----  Node 3 (PMEM)
		  40

Quoted 6 lines

node distances:
node 0 1 2 3

   0  10   20   30   40
   1  20   10   40   30
   2  30   40   10   40
   3  40   30   40   10  

I'm fairly confused by this example. Do all nodes have CPUs? Isn't this just classic NUMA, where optimizing for locality makes the most sense, rather than tiering?

Forget the interface for a second, I have no idea how tiering on such a system would work. One CPU's lower tier can be another CPU's toptier. There is no lowest rung from which to actually *reclaim* pages. Would the CPUs just demote in circles?

And the coldest pages on one socket would get demoted into another socket and displace what that socket considers hot local memory?

I feel like I missing something.

When we're talking about tiered memory, I'm thinking about CPUs utilizing more than one memory node. If those other nodes have CPUs, you can't reliably establish a singular tier order anymore and it becomes classic NUMA, no?

Quoted 85 lines · On Thu, 16 Jun 2022 09:11:24 +0800 Ying Huang <[email protected]> wrote:

Quoted 84 lines · Johannes Weiner

Quoted 29 lines · Aneesh Kumar K V

Quoted 10 lines · Johannes Weiner

Quoted 8 lines · Aneesh Kumar K V

If the kernel still can't make the right decision, userspace could rearrange them in any order using rank values. Without something like rank, if userspace needs to fix things up, it gets hard with device hotplugging. ie, the userspace policy could be that any new PMEM tier device that is hotplugged, park it with a very low-rank value and hence lowest in demotion order by default. (echo 10 > /sys/devices/system/memtier/memtier2/rank) . After that userspace could selectively move the new devices to the correct memory tier?

Quoted 8 lines

I had touched on this in the other email.

This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface.

Considering the kernel has a static map for these tiers, how can two drivers end up using the same tier? If a new driver is going to manage a memory device that is of different characteristics than the one managed by dax/kmem, we will end up adding

#define MEMORY_TIER_NEW_DEVICE 4

The new driver will never use MEMORY_TIER_PMEM

Quoted 7 lines

What can happen is two devices that are managed by DAX/kmem that should be in two memory tiers get assigned the same memory tier because the dax/kmem driver added both the device to the same memory tier.

In the future we would avoid that by using more device properties like HMAT to create additional memory tiers with different rank values. ie, we would do in the dax/kmem create_tier_from_rank() .

Yes, that's the type of collision I mean. Two GPUs, two CXL-attached DRAMs of different speeds etc.

I also like Huang's idea of using latency characteristics instead of abstract distances. Though I'm not quite sure how feasible this is in the short term, and share some concerns that Jonathan raised. But I think a wider possible range to begin with makes sense in any case.

Quoted 11 lines

Quoted 4 lines

In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation.

I understand that the driver override will be done via module parameters. How will we implement device override? For example in case of dax/kmem driver the device override will be per dax device? What interface will we use to set the override?

IIUC in the above proposal the dax/kmem will do

Quoted 8 lines

node_create_and_set_memory_tier(numa_node, get_device_tier_index(dev_dax));

get_device_tier_index(struct dev_dax *dev)
{
 return dax_kmem_tier_index; // module parameter
}

Are you suggesting to add a dev_dax property to override the tier defaults?

I was thinking a new struct memdevice and struct memtype(?). Every driver implementing memory devices like this sets those up and registers them with generic code and preset parameters. The generic code creates sysfs directories and allows overriding the parameters.

struct memdevice {
	struct device dev;
	unsigned long distance;
	struct list_head siblings;
	/* nid? ... */
};

struct memtype {
	struct device_type type;
	unsigned long default_distance;
	struct list_head devices;
};

Quoted 7 lines

Quoted 1 line

That forms the (tweakable) tree describing physical properties.

In general, I think memtype is a good idea. I have suggested something similar before. It can describe the characters of a specific type of memory (same memory media with different interface (e.g., CXL, or DIMM) will be different memory types). And they can be used to provide overriding information.

I'm not sure you are suggesting interface as one element of distinguishing types, or as the element - just in case it's as 'the element'. Ignore the next bit if not ;)

Memory "interface" isn't going to be enough of a distinction. If you want to have a default distance it would need to be different for cases where the same 'type' of RAM has very different characteristics. Applies everywhere but given CXL 'defines' a lot of this - if we just have DRAM attached via CXL:

1. 16-lane direct attached DRAM device. (low latency - high bw)
2. 4x 16-lane direct attached DRAM interleaved (low latency - very high bw)
3. 4-lane direct attached DRAM device (low latency - low bandwidth)
4. 16-lane to single switch, 4x 4-lane devices interleaved (mid latency - high bw)
5. 4-lane to single switch, 4x 4-lane devices interleaved (mid latency, mid bw)
6. 4x 16-lane so 4 switch, each switch to 4 DRAM devices (mid latency, very high bw) (7. 16 land directed attached nvram. (midish latency, high bw - perf wise might be similarish to 4).

It could be a lot more complex, but hopefully that conveys that 'type' is next to useless to characterize things unless we have a very large number of potential subtypes. If we were on current tiering proposal we'd just have the CXL subsystem manage multiple tiers to cover what is attached.

Quoted 68 lines

As for memdevice, I think that we already have "node" to represent them in sysfs. Do we really need another one? Is it sufficient to add some links to node in the appropriate directory? For example, make memtype class device under the physical device (e.g. CXL device), and create links to node inside the memtype class device directory?

Quoted 1 line

From that, the kernel then generates the ordered list of tiers.

As Jonathan Cameron pointed, we may need the memory tier ID to be stable if possible. I know this isn't a easy task. At least we can make the default memory tier (CPU local DRAM) ID stable (for example make it always 128)? That provides an anchor for users to understand.

Best Regards,
Huang, Ying

Quoted 49 lines

Quoted 24 lines

Quoted 4 lines

This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers.

Can you elaborate more on how distance value will be used? The device/device NUMA node can have different distance value from other NUMA nodes. How do we group them?
for ex: earlier discussion did outline three different topologies. Can you ellaborate how we would end up grouping them using distance?

For ex: in the topology below node 2 is at distance 30 from Node0 and 40 from Nodes so how will we classify node 2?

Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes.

		  20
Node 0 (DRAM)  ----  Node 1 (DRAM)
 |        \   /       |
 | 30    40 X 40      | 30
 |        /   \       |
Node 2 (PMEM)  ----  Node 3 (PMEM)
		  40

Quoted 6 lines

node distances:
node 0 1 2 3

0  10   20   30   40
1  20   10   40   30
2  30   40   10   40
3  40   30   40   10  

I'm fairly confused by this example. Do all nodes have CPUs? Isn't this just classic NUMA, where optimizing for locality makes the most sense, rather than tiering?

Forget the interface for a second, I have no idea how tiering on such a system would work. One CPU's lower tier can be another CPU's toptier. There is no lowest rung from which to actually *reclaim* pages. Would the CPUs just demote in circles?

And the coldest pages on one socket would get demoted into another socket and displace what that socket considers hot local memory?

I feel like I missing something.

When we're talking about tiered memory, I'm thinking about CPUs utilizing more than one memory node. If those other nodes have CPUs, you can't reliably establish a singular tier order anymore and it becomes classic NUMA, no?

Quoted 8 lines · Jonathan Cameron

Quoted 5 lines · On Thu, 9 Jun 2022 16:41:04 -0400 Johannes Weiner <[email protected]> wrote:

On Thu, Jun 09, 2022 at 03:22:43PM +0100, Jonathan Cameron wrote:
Would it make more sense to have the platform/devicetree/driver provide more fine-grained distance values similar to NUMA distances, and have a driver-scope tunable to override/correct? And then have the distance value function as the unique tier ID and rank in one.

Quoted 10 lines

Absolutely a good thing to provide that information, but it's black magic. There are too many contradicting metrics (latency vs bandwidth etc) even not including a more complex system model like Jerome Glisse proposed a few years back. https://lore.kernel.org/all/[email protected]/
CXL 2.0 got this more right than anything else I've seen as provides discoverable topology along with details like latency to cross between particular switch ports. Actually using that data (other than by throwing it to userspace controls for HPC apps etc) is going to take some figuring out. Even the question of what + how we expose this info to userspace is non obvious.

Right, I don't think those would be scientifically accurate - but neither is a number between 1 and 3.

The way I look at it is more
about spreading out the address space a bit, to allow expressing nuanced differences without risking conflicts and overlaps. Hopefully this results in the shipped values stabilizing over time and thus requiring less and less intervention and overriding from userspace.

Quoted 25 lines

Quoted 9 lines

Going further, it could be useful to separate the business of hardware properties (and configuring quirks) from the business of configuring MM policies that should be applied to the resulting tier hierarchy. They're somewhat orthogonal tuning tasks, and one of them might become obsolete before the other (if the quality of distance values provided by drivers improves before the quality of MM heuristics ;). Separating them might help clarify the interface for both designers and users.

Quoted 4 lines

E.g. a memdev class scope with a driver-wide distance value, and a memdev scope for per-device values that default to "inherit driver value". The memtier subtree would then have an r/o structure, but allow tuning per-tier interleaving ratio[1], demotion rules etc.

Ok that makes sense. I'm not sure if that ends up as an implementation detail, or effects the userspace interface of this particular element.

I'm not sure completely read only is flexible enough (though mostly RO is fine) as we keep sketching out cases where any attempt to do things automatically does the wrong thing and where we need to add an extra tier to get everything to work. Short of having a lot of tiers I'm not sure how we could have the default work well. Maybe a lot of "tiers" is fine though perhaps we need to rename them if going this way and then they don't really work as current concept of tier.

Quoted 5 lines

Imagine a system with subtle difference between different memories such as 10% latency increase for same bandwidth. To get an advantage from demoting to such a tier will require really stable usage and long run times. Whilst you could design a demotion scheme that takes that into account, I think we are a long way from that today.

Good point: there can be a clear hardware difference, but it's a policy choice whether the MM should treat them as one or two tiers.

What do you think of a per-driver/per-device (overridable) distance number, combined with a configurable distance cutoff for what constitutes separate tiers. E.g. cutoff=20 means two devices with distances of 10 and 20 respectively would be in the same tier, devices with 10 and 100 would be in separate ones. The kernel then generates and populates the tiers based on distances and grouping cutoff, and populates the memtier directory tree and nodemasks in sysfs.

It could be simple tier0, tier1, tier2 numbering again, but the numbers now would mean something to the user. A rank tunable is no longer necessary.

I think even the nodemasks in the memtier tree could be read-only then, since corrections should only be necessary when either the device distance is wrong or the tier grouping cutoff.

Can you think of scenarios where that scheme would fall apart?

Quoted 4 lines

It could be simple tier0, tier1, tier2 numbering again, but the numbers now would mean something to the user. A rank tunable is no longer necessary.

This feels like it might make tier assignments a bit less stable and hence run into question of how to hook up accounting. Not my area of expertise though, but it was put forward as one of the reasons we didn't want hotplug to potentially end up shuffling other tiers around. The desire was for a 'stable' entity. Can avoid that with 'space' between them but then we sort of still have rank, just in a form that makes updating it messy (need to create a new tier to do it).

Quoted 1 line

mm/memory-tiers.c static const struct attribute_group *memory_tier_attr_groups[] = { · line 168

Quoted 7 lines · Aneesh Kumar K.V

mm/memory-tiers.c +5 -0

+static struct memory_tier *__node_get_memory_tier(int node)
+{
+	struct memory_tier *memtier;
+
+	list_for_each_entry(memtier, &memory_tiers, list) {

We could need to map node to mem_tier quite often, if we need to account memory usage at tier level. It will be more efficient to have a pointer from node (pgdat) to memtier rather than doing a search through the list.

mm/memory-tiers.c static const struct attribute_group *memory_tier_attr_groups[] = { · line 174

Quoted 7 lines

mm/memory-tiers.c +6 -0

+		if (node_isset(node, memtier->nodelist))
+			return memtier;
+	}
+	return NULL;
+}
+

Tim

Quoted 6 lines · Tim Chen

Quoted 5 lines · Aneesh Kumar K.V

Quoted 12 lines

mm/memory-tiers.c static const struct attribute_group *memory_tier_attr_groups[] = { · line 168

Quoted 5 lines

mm/memory-tiers.c +5 -0

+static struct memory_tier *__node_get_memory_tier(int node)
+{
+	struct memory_tier *memtier;
+
+	list_for_each_entry(memtier, &memory_tiers, list) {

We could need to map node to mem_tier quite often, if we need to account memory usage at tier level. It will be more efficient to have a pointer from node (pgdat) to memtier rather than doing a search through the list.

That is something I was actively trying to avoid. Currently all struct memory_tier references are with memory_tier_lock mutex held. That simplify the locking and reference counting.

As of now we are able to implement all the required interfaces without pgdat having pointers to struct memory_tier. We can update pgdat with memtier details when we are implementing changes requiring those. We could keep additional memtier->dev reference to make sure memory tiers are not destroyed while other part of the kernel is referencing the same. But IMHO such changes should wait till we have users for the same.

Quoted 10 lines

mm/memory-tiers.c static const struct attribute_group *memory_tier_attr_groups[] = { · line 174

Quoted 7 lines

mm/memory-tiers.c +6 -0

+		if (node_isset(node, memtier->nodelist))
+			return memtier;
+	}
+	return NULL;
+}
+

Tim

-aneesh

Quoted 14 lines · Tim Chen

mm/memory-tiers.c static const struct attribute_group *memory_tier_attr_groups[] = { · line 168

Quoted 6 lines · Aneesh Kumar K.V

mm/memory-tiers.c +5 -0

+static struct memory_tier *__node_get_memory_tier(int node)
+{
+	struct memory_tier *memtier;
+
+	list_for_each_entry(memtier, &memory_tiers, list) {

We could need to map node to mem_tier quite often, if we need to account memory usage at tier level. It will be more efficient to have a pointer from node (pgdat) to memtier rather than doing a search through the list.

That is something I was actively trying to avoid. Currently all struct memory_tier references are with memory_tier_lock mutex held. That simplify the locking and reference counting.

As of now we are able to implement all the required interfaces without pgdat having pointers to struct memory_tier. We can update pgdat with memtier details when we are implementing changes requiring those. We could keep additional memtier->dev reference to make sure memory tiers are not destroyed while other part of the kernel is referencing the same. But IMHO such changes should wait till we have users for the same.

Quoted 20 lines · Aneesh Kumar K V

Quoted 14 lines · Tim Chen

mm/memory-tiers.c static const struct attribute_group *memory_tier_attr_groups[] = { · line 168

Quoted 6 lines · Aneesh Kumar K.V

mm/memory-tiers.c +5 -0

+static struct memory_tier *__node_get_memory_tier(int node)
+{
+	struct memory_tier *memtier;
+
+	list_for_each_entry(memtier, &memory_tiers, list) {

We could need to map node to mem_tier quite often, if we need to account memory usage at tier level. It will be more efficient to have a pointer from node (pgdat) to memtier rather than doing a search through the list.

That is something I was actively trying to avoid. Currently all struct memory_tier references are with memory_tier_lock mutex held. That simplify the locking and reference counting.

Quoted 7 lines

As of now we are able to implement all the required interfaces without pgdat having pointers to struct memory_tier. We can update pgdat with memtier details when we are implementing changes requiring those. We could keep additional memtier->dev reference to make sure memory tiers are not destroyed while other part of the kernel is referencing the same. But IMHO such changes should wait till we have users for the same.

I think we should have an efficient mapping from node to memtier from the get go. There are many easily envisioned scenarios where we need to map from node to memtier, which Ying pointed out.