mm/demotion: Expose memory tier details via sysfs

PATCH Superseded Aneesh Kumar K.V 3 participants 25 messages Aug 30, 2022 – Sep 5, 2022

Subsystems (3)

Memory management MM misc MM NUMA placement

lore

Share & tools

lore

Aneesh Kumar K.V

author

2022-08-30 08:18 UTC #0 lore raw

PATCH v3 1/1 mm/demotion: Expose memory tier details via sysfs

This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/

The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes

A directory hierarchy looks like

:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent

All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes

:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2

Signed-off-by Aneesh Kumar K.V <[email protected]>

---

Changes from v2:
* update macro to static inline
* Fix build error with CONFIG_MIGRATION disabled
* drop abstract_distance
* update commit message

 .../ABI/testing/sysfs-kernel-mm-memory-tiers  |  35 ++++
 mm/memory-tiers.c                             | 154 +++++++++++++++---
 2 files changed, 167 insertions(+), 22 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers

Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers new file +35 -0

new file mode 100644
index 000000000000..55051fcf5502
@@ -0,0 +1,35 @@
+What:		/sys/devices/virtual/memory_tiering/
+Date:		August 2022
+Contact:	Linux memory management mailing list <[email protected]>
+Description:	A collection of all the memory tiers allocated.
+
+		Individual memory tier details are contained in subdirectories
+		named by the abstract distance of the memory tier.
+
+		/sys/devices/virtual/memory_tiering/memory_tierN/
+
+
+What:		/sys/devices/virtual/memory_tiering/memory_tierN/
+		/sys/devices/virtual/memory_tiering/memory_tierN/nodes
+Date:		August 2022
+Contact:	Linux memory management mailing list <[email protected]>
+Description:	Directory with details of a specific memory tier
+
+		This is the directory containing information about a particular
+		memory tier, memtierN, where N is derived based on abstract distance.
+
+		A smaller value of N implies a higher (faster) memory tier in the
+		hierarchy.
+
+		nodes: NUMA nodes that are part of this memory tier.
+
+
+What:		/sys/devices/virtual/memory_tiering/toptier_nodes
+Date:		August 2022
+Contact:	Linux memory management mailing list <[email protected]>
+Description:	Toptier node mask
+
+		A toptier is defined as the memory tier from which memory promotion
+		is not done by the kernel.
+
+		toptier_nodes: Union of NUMA nodes that are part of each toptier.

mm/memory-tiers.c +132 ~~-22~~

214 lines not on this page · raw message

Signature · 1 line

2.37.2

Huang, Ying

2022-09-01 07:01 UTC #1 lore raw

Quoted 6 lines · Aneesh Kumar K.V

This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes

I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.

Quoted 8 lines

A directory hierarchy looks like

:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent

Quoted 7 lines

All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes
:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2

I don't think that it is a good idea to show toptier information in user space interface. Because it is just a in kernel implementation details. Now, we only promote pages from !toptier to toptier. But there may be multiple memory tiers in toptier and !toptier, we may change the implementation in the future. For example, we may promote pages from DRAM to HBM in the future.

Do we need a way to show the default memory tier in sysfs? That is, the memory tier that the DRAM nodes belong to.

Best Regards,
Huang, Ying

Quoted 10 lines

Signed-off-by Aneesh Kumar K.V <[email protected]>
---
Changes from v2:
* update macro to static inline
* Fix build error with CONFIG_MIGRATION disabled
* drop abstract_distance
* update commit message

[snip]

Aneesh Kumar K V

author

2022-09-01 08:25 UTC #2 lore raw

Quoted 30 lines · Ying

Quoted 6 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 8 lines
A directory hierarchy looks like
:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent
Quoted 7 lines
All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes
:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2

Quoted 7 lines

I don't think that it is a good idea to show toptier information in user space interface. Because it is just a in kernel implementation details. Now, we only promote pages from !toptier to toptier. But there may be multiple memory tiers in toptier and !toptier, we may change the implementation in the future. For example, we may promote pages from DRAM to HBM in the future.

In the case you describe above and others, we will always have a list of NUMA nodes from which memory promotion is not done. /sys/devices/virtual/memory_tiering/toptier_nodes shows that list.

Quoted 3 lines

Do we need a way to show the default memory tier in sysfs? That is, the memory tier that the DRAM nodes belong to.

I will hold adding that until we have support for modifying memory tier details from userspace. That is when userspace would want to know about the default memory tier.

For now, the user interface is a simpler hierarchy of memory tiers, it's associated nodes and the list of nodes from which promotion is not done.

-aneesh

Huang, Ying

2022-09-02 00:33 UTC #3 lore raw

Quoted 40 lines · Aneesh Kumar K V

Quoted 30 lines · Ying
Quoted 6 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 8 lines
A directory hierarchy looks like
:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent
Quoted 7 lines
All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes
:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2
Quoted 7 lines
I don't think that it is a good idea to show toptier information in user space interface. Because it is just a in kernel implementation details. Now, we only promote pages from !toptier to toptier. But there may be multiple memory tiers in toptier and !toptier, we may change the implementation in the future. For example, we may promote pages from DRAM to HBM in the future.

Quoted 3 lines

In the case you describe above and others, we will always have a list of NUMA nodes from which memory promotion is not done. /sys/devices/virtual/memory_tiering/toptier_nodes shows that list.

I don't think we will need that interface if we don't restrict promotion in the future. For example, he can just check the memory tier with smallest number.

TBH, I don't know why do we need that interface. What is it for? We don't want to expose unnecessary information to restrict our in kernel implementation in the future.

So, please remove that interface at least before we discussing it thoroughly.

Quoted 9 lines

Quoted 3 lines
Do we need a way to show the default memory tier in sysfs? That is, the memory tier that the DRAM nodes belong to.
I will hold adding that until we have support for modifying memory tier details from userspace. That is when userspace would want to know about the default memory tier.
For now, the user interface is a simpler hierarchy of memory tiers, it's associated nodes and the list of nodes from which promotion is not done.

OK.

Best Regards,
Huang, Ying

Wei Xu

2022-09-02 05:09 UTC #4 lore raw

Quoted 51 lines · Ying

Quoted 40 lines · Aneesh Kumar K V
Quoted 30 lines · Ying
Quoted 6 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 8 lines
A directory hierarchy looks like
:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent
Quoted 7 lines
All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes
:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2
Quoted 7 lines
I don't think that it is a good idea to show toptier information in user space interface. Because it is just a in kernel implementation details. Now, we only promote pages from !toptier to toptier. But there may be multiple memory tiers in toptier and !toptier, we may change the implementation in the future. For example, we may promote pages from DRAM to HBM in the future.
Quoted 3 lines
In the case you describe above and others, we will always have a list of NUMA nodes from which memory promotion is not done. /sys/devices/virtual/memory_tiering/toptier_nodes shows that list.
I don't think we will need that interface if we don't restrict promotion in the future. For example, he can just check the memory tier with smallest number.

Quoted 6 lines

TBH, I don't know why do we need that interface. What is it for? We don't want to expose unnecessary information to restrict our in kernel implementation in the future.
So, please remove that interface at least before we discussing it thoroughly.

I have asked for this interface to allow the userspace to query a list of top-tier nodes as the targets of userspace-driven promotions. The idea is that demotion can gradually go down tier by tier, but we promote hot pages directly to the top-tier and bypass the immediate tiers.

Certainly, this can be viewed as a policy choice. Given that now we have a clearly defined memory tier hierarchy in sysfs and the toptier_nodes content can be constructed from this memory tier hierarchy and other information from the node sysfs interfaces, I am fine if we want to remove toptier_nodes and keep the current memory tier sysfs interfaces to the minimal.

Wei Xu

Quoted 14 lines

Quoted 9 lines
Quoted 3 lines
Do we need a way to show the default memory tier in sysfs? That is, the memory tier that the DRAM nodes belong to.
I will hold adding that until we have support for modifying memory tier details from userspace. That is when userspace would want to know about the default memory tier.
For now, the user interface is a simpler hierarchy of memory tiers, it's associated nodes and the list of nodes from which promotion is not done.
OK.
Best Regards,
Huang, Ying

Huang, Ying

2022-09-02 05:15 UTC #5 lore raw

Quoted 59 lines · Wei Xu

Quoted 51 lines · Ying
Quoted 40 lines · Aneesh Kumar K V
Quoted 30 lines · Ying
Quoted 6 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 8 lines
A directory hierarchy looks like
:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent
Quoted 7 lines
All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes
:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2
Quoted 7 lines
I don't think that it is a good idea to show toptier information in user space interface. Because it is just a in kernel implementation details. Now, we only promote pages from !toptier to toptier. But there may be multiple memory tiers in toptier and !toptier, we may change the implementation in the future. For example, we may promote pages from DRAM to HBM in the future.
Quoted 3 lines
In the case you describe above and others, we will always have a list of NUMA nodes from which memory promotion is not done. /sys/devices/virtual/memory_tiering/toptier_nodes shows that list.
I don't think we will need that interface if we don't restrict promotion in the future. For example, he can just check the memory tier with smallest number.
Quoted 6 lines
TBH, I don't know why do we need that interface. What is it for? We don't want to expose unnecessary information to restrict our in kernel implementation in the future.
So, please remove that interface at least before we discussing it thoroughly.

Quoted 7 lines

I have asked for this interface to allow the userspace to query a list of top-tier nodes as the targets of userspace-driven promotions. The idea is that demotion can gradually go down tier by tier, but we promote hot pages directly to the top-tier and bypass the immediate tiers.
Certainly, this can be viewed as a policy choice.

Yes. It's possible for us to change this in the future.

Quoted 5 lines

Given that now we have a clearly defined memory tier hierarchy in sysfs and the toptier_nodes content can be constructed from this memory tier hierarchy and other information from the node sysfs interfaces, I am fine if we want to remove toptier_nodes and keep the current memory tier sysfs interfaces to the minimal.

Thanks!

Best Regards,
Huang, Ying

Quoted 14 lines

Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
Do we need a way to show the default memory tier in sysfs? That is, the memory tier that the DRAM nodes belong to.
I will hold adding that until we have support for modifying memory tier details from userspace. That is when userspace would want to know about the default memory tier.
For now, the user interface is a simpler hierarchy of memory tiers, it's associated nodes and the list of nodes from which promotion is not done.
OK.
Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-02 05:24 UTC #6 lore raw

In reply to Wei Xu

Quoted 11 lines · Wei Xu

Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.

I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion

https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com

Quoted 46 lines

Quoted 33 lines
Quoted 25 lines
Quoted 16 lines
Quoted 8 lines
A directory hierarchy looks like
:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent
Quoted 7 lines
All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes
:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2
Quoted 7 lines
I don't think that it is a good idea to show toptier information in user space interface. Because it is just a in kernel implementation details. Now, we only promote pages from !toptier to toptier. But there may be multiple memory tiers in toptier and !toptier, we may change the implementation in the future. For example, we may promote pages from DRAM to HBM in the future.
Quoted 3 lines
In the case you describe above and others, we will always have a list of NUMA nodes from which memory promotion is not done. /sys/devices/virtual/memory_tiering/toptier_nodes shows that list.
I don't think we will need that interface if we don't restrict promotion in the future. For example, he can just check the memory tier with smallest number.
Quoted 6 lines
TBH, I don't know why do we need that interface. What is it for? We don't want to expose unnecessary information to restrict our in kernel implementation in the future.
So, please remove that interface at least before we discussing it thoroughly.
I have asked for this interface to allow the userspace to query a list of top-tier nodes as the targets of userspace-driven promotions. The idea is that demotion can gradually go down tier by tier, but we promote hot pages directly to the top-tier and bypass the immediate tiers.

Quoted 7 lines

Certainly, this can be viewed as a policy choice. Given that now we have a clearly defined memory tier hierarchy in sysfs and the toptier_nodes content can be constructed from this memory tier hierarchy and other information from the node sysfs interfaces, I am fine if we want to remove toptier_nodes and keep the current memory tier sysfs interfaces to the minimal.

Ok I can do a v4 with toptier_nodes dropped.

-aneesh

Huang, Ying

2022-09-02 05:40 UTC #7 lore raw

Quoted 22 lines · Aneesh Kumar K V

Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.

Quoted 5 lines

I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com

I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.

Quoted 55 lines

Quoted 46 lines
Quoted 33 lines
Quoted 25 lines
Quoted 16 lines
Quoted 8 lines
A directory hierarchy looks like
:/sys/devices/virtual/memory_tiering$ tree memory_tier4/
memory_tier4/
├── nodes
├── subsystem -> ../../../../bus/memory_tiering
└── uevent
Quoted 7 lines
All toptier nodes are listed via
/sys/devices/virtual/memory_tiering/toptier_nodes
:/sys/devices/virtual/memory_tiering$ cat toptier_nodes
0,2
:/sys/devices/virtual/memory_tiering$ cat memory_tier4/nodes
0,2
Quoted 7 lines
I don't think that it is a good idea to show toptier information in user space interface. Because it is just a in kernel implementation details. Now, we only promote pages from !toptier to toptier. But there may be multiple memory tiers in toptier and !toptier, we may change the implementation in the future. For example, we may promote pages from DRAM to HBM in the future.
Quoted 3 lines
In the case you describe above and others, we will always have a list of NUMA nodes from which memory promotion is not done. /sys/devices/virtual/memory_tiering/toptier_nodes shows that list.
I don't think we will need that interface if we don't restrict promotion in the future. For example, he can just check the memory tier with smallest number.
Quoted 6 lines
TBH, I don't know why do we need that interface. What is it for? We don't want to expose unnecessary information to restrict our in kernel implementation in the future.
So, please remove that interface at least before we discussing it thoroughly.
I have asked for this interface to allow the userspace to query a list of top-tier nodes as the targets of userspace-driven promotions. The idea is that demotion can gradually go down tier by tier, but we promote hot pages directly to the top-tier and bypass the immediate tiers.
Quoted 7 lines
Certainly, this can be viewed as a policy choice. Given that now we have a clearly defined memory tier hierarchy in sysfs and the toptier_nodes content can be constructed from this memory tier hierarchy and other information from the node sysfs interfaces, I am fine if we want to remove toptier_nodes and keep the current memory tier sysfs interfaces to the minimal.

Quoted 1 line

Ok I can do a v4 with toptier_nodes dropped.

Thanks!

Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-02 05:46 UTC #8 lore raw

Quoted 28 lines · Ying

Quoted 26 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion

Quoted 6 lines

Quoted 1 line
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.

All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .

-aneesh

Huang, Ying

2022-09-02 06:12 UTC #9 lore raw

Quoted 31 lines · Aneesh Kumar K V

Quoted 30 lines · Ying
Quoted 27 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com

Quoted 8 lines

Quoted 4 lines
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .

IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.

Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-02 06:32 UTC #10 lore raw

Quoted 38 lines · Ying

Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.

Quoted 8 lines

Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.

Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.

/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN

-aneesh

Huang, Ying

2022-09-02 06:40 UTC #11 lore raw

Quoted 48 lines · Aneesh Kumar K V

Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.

Quoted 6 lines

Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN

I think we should add

/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN

I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.

Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-02 06:44 UTC #12 lore raw

Quoted 54 lines · Ying

Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.

Quoted 8 lines

Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN

I am trying to find if there is a technical reason to do the same?

Quoted 4 lines

I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.

I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).

-aneesh

Wei Xu

2022-09-02 07:02 UTC #13 lore raw

Quoted 72 lines · Aneesh Kumar K V

Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier related details can be found. All allocated memory tiers will be listed there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via /sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.

Quoted 6 lines

I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh

Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):

ls -1 /sys/bus/nd/devices

namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3

So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.

Wei

Huang, Ying

2022-09-02 07:58 UTC #14 lore raw

Quoted 96 lines · Wei Xu

Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
Quoted 4 lines · Aneesh Kumar K.V
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than memory_tiering. Because we have a set of memory_tierN devices inside. "memory_tier" sounds more natural. I know this is subjective, just my preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3

Quoted 2 lines

So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.

Thanks for pointing this out. My original understanding of driver core isn't correct.

But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.

Just my 2 cents.

Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-02 08:49 UTC #15 lore raw

Quoted 104 lines · Ying

Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
Quoted 6 lines · Ying
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.

Quoted 7 lines

But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.

The latency and bandwidth details are already exposed via

/sys/devices/system/node/nodeY/access0/initiators/

Documentation/admin-guide/mm/numaperf.rst

That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth

-aneesh

Huang, Ying

2022-09-02 09:05 UTC #16 lore raw

Quoted 113 lines · Aneesh Kumar K V

Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Quoted 7 lines · Aneesh Kumar K V
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.

Quoted 8 lines

The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth

Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.

Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-02 09:44 UTC #17 lore raw

Quoted 121 lines · Ying

Quoted 119 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 11 lines · Wei Xu
Quoted 10 lines · Ying
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
Quoted 8 lines
Quoted 8 lines
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst

Quoted 8 lines

Quoted 2 lines
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.

I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.

We might end up adding memory_type there if we allow changing "abstract distance" of a memory type from userspace later. Otherwise, I don't see a reason for memory type to be exposed. But then we don't have to decide on this now.

-aneesh

Huang, Ying

2022-09-05 01:53 UTC #18 lore raw

Quoted 131 lines · Aneesh Kumar K V

Quoted 121 lines · Ying
Quoted 119 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 22 lines · Aneesh Kumar K V
Quoted 19 lines · Wei Xu
On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <[email protected]> wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
Quoted 5 lines
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 because we would want it to a susbsystem where all memory tiering related details can be found including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
Quoted 8 lines
Quoted 2 lines
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.

Quoted 3 lines

I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.

I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types.

Quoted 3 lines

We might end up adding memory_type there if we allow changing "abstract distance" of a memory type from userspace later. Otherwise, I don't see a reason for memory type to be exposed. But then we don't have to decide on this now.

As above, because I think memory_type can provide value even outside of memory_tier, I prefer to add memory_type sysfs interface anyway personally.

Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-05 03:50 UTC #19 lore raw

Quoted 133 lines · Ying

Quoted 131 lines · Aneesh Kumar K V
Quoted 129 lines · Ying
Quoted 113 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Quoted 27 lines · Aneesh Kumar K V
On 9/2/22 10:39 AM, Wei Xu wrote:
On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <[email protected]> wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
because we would want it to a susbsystem where all memory tiering related details can be found
including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one subsystem (bus). If my understanding were correct, that breaks the driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
Quoted 8 lines
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.

Quoted 9 lines

Quoted 3 lines
I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.
I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types.

IMHO we can decide on that based on why we end up adding memory_type details to sysfs. If that is only for memory tier modification from userspace we can look at adding that in the memory tiering sysfs hierarchy.

Also since we have precedence of consolidating things within a sysfs hierarchy as explained in previous emails, I think we should keep "memory_tiering" as sysfs subsystem name? I hope we can get an agreement on that for now?

-aneesh

Huang, Ying

2022-09-05 05:14 UTC #20 lore raw

Quoted 143 lines · Aneesh Kumar K V

Quoted 133 lines · Ying
Quoted 131 lines · Aneesh Kumar K V
Quoted 129 lines · Ying
Quoted 113 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
Quoted 34 lines · Ying
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 10:39 AM, Wei Xu wrote:
On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <[email protected]> wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
because we would want it to a susbsystem where all memory tiering related details can be found
including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one
subsystem (bus). If my understanding were correct, that breaks the
driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
Quoted 8 lines
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.
Quoted 9 lines
Quoted 3 lines
I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.
I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types.

Quoted 7 lines

IMHO we can decide on that based on why we end up adding memory_type details to sysfs. If that is only for memory tier modification from userspace we can look at adding that in the memory tiering sysfs hierarchy.
Also since we have precedence of consolidating things within a sysfs hierarchy as explained in previous emails, I think we should keep "memory_tiering" as sysfs subsystem name? I hope we can get an agreement on that for now?

I prefer to separate memory_tier and memory_type, so the subsystem name should be "memory_tier". You prefer to consolidate memory_tier and memory_type, so the subsystem name should be "memory_tiering".

The main reason behind my idea is that memory_type isn't tied with memory tiering directly. It describes some hardware property. Even if we don't use memory tiering, we can still use that to classify the memory devices in the system.

Why do you want to consolidate them? To reduce one directory from sysfs?

I want to get opinions from other people too.

Best Regards,
Huang, Ying

Aneesh Kumar K V

author

2022-09-05 05:28 UTC #21 lore raw

Quoted 157 lines · Ying

Quoted 143 lines · Aneesh Kumar K V
Quoted 133 lines · Ying
Quoted 131 lines · Aneesh Kumar K V
Quoted 129 lines · Ying
Quoted 113 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 38 lines · Ying
Quoted 36 lines · Aneesh Kumar K V
On 9/2/22 11:10 AM, Huang, Ying wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 10:39 AM, Wei Xu wrote:
On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <[email protected]> wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
because we would want it to a susbsystem where all memory tiering related details can be found
including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one
subsystem (bus). If my understanding were correct, that breaks the
driver core convention.
Quoted 8 lines
Quoted 3 lines
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices.
memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows
details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have almost totally different attributes (sysfs file). So, we should create 2 buses for them. Each has its own attribute group. "virtual" itself isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
Quoted 8 lines
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.
Quoted 9 lines
Quoted 3 lines
I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.
I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types.
Quoted 7 lines
IMHO we can decide on that based on why we end up adding memory_type details to sysfs. If that is only for memory tier modification from userspace we can look at adding that in the memory tiering sysfs hierarchy.
Also since we have precedence of consolidating things within a sysfs hierarchy as explained in previous emails, I think we should keep "memory_tiering" as sysfs subsystem name? I hope we can get an agreement on that for now?
I prefer to separate memory_tier and memory_type, so the subsystem name should be "memory_tier". You prefer to consolidate memory_tier and memory_type, so the subsystem name should be "memory_tiering".

Quoted 8 lines

The main reason behind my idea is that memory_type isn't tied with memory tiering directly. It describes some hardware property. Even if we don't use memory tiering, we can still use that to classify the memory devices in the system.
Why do you want to consolidate them? To reduce one directory from sysfs?

So that it is much intuitive for user to got to memory_tiering sysfs hierarchy to change the memory tier levels. As I mentioned earlier the reason for consolidating things is to accommodate the possibility of supporting changing abstract distance of a memory type so that we can change the memory tier assignment of that specific memory type. I don't see any other reason we would want to expose memory type to userspace as of now.

Quoted 4 lines

I want to get opinions from other people too.
Best Regards,
Huang, Ying

-aneesh

Huang, Ying

2022-09-05 05:54 UTC #22 lore raw

Quoted 167 lines · Aneesh Kumar K V

Quoted 157 lines · Ying
Quoted 143 lines · Aneesh Kumar K V
Quoted 133 lines · Ying
Quoted 131 lines · Aneesh Kumar K V
Quoted 129 lines · Ying
Quoted 113 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
Quoted 46 lines · Ying
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 11:10 AM, Huang, Ying wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 10:39 AM, Wei Xu wrote:
On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <[email protected]> wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
because we would want it to a susbsystem where all memory tiering related details can be found
including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one
subsystem (bus). If my understanding were correct, that breaks the
driver core convention.
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices.
memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows
details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have
almost totally different attributes (sysfs file). So, we should create
2 buses for them. Each has its own attribute group. "virtual" itself
isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate them within the same subdirectory? I am still not clear why you are suggesting they need to be in different sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
Quoted 8 lines
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.
Quoted 9 lines
Quoted 3 lines
I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.
I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types.
Quoted 7 lines
IMHO we can decide on that based on why we end up adding memory_type details to sysfs. If that is only for memory tier modification from userspace we can look at adding that in the memory tiering sysfs hierarchy.
Also since we have precedence of consolidating things within a sysfs hierarchy as explained in previous emails, I think we should keep "memory_tiering" as sysfs subsystem name? I hope we can get an agreement on that for now?
I prefer to separate memory_tier and memory_type, so the subsystem name should be "memory_tier". You prefer to consolidate memory_tier and memory_type, so the subsystem name should be "memory_tiering".
Quoted 8 lines
The main reason behind my idea is that memory_type isn't tied with memory tiering directly. It describes some hardware property. Even if we don't use memory tiering, we can still use that to classify the memory devices in the system.
Why do you want to consolidate them? To reduce one directory from sysfs?

Quoted 5 lines

So that it is much intuitive for user to got to memory_tiering sysfs hierarchy to change the memory tier levels. As I mentioned earlier the reason for consolidating things is to accommodate the possibility of supporting changing abstract distance of a memory type so that we can change the memory tier assignment of that specific
memory type.

If we put memory_tier and memory_type into 2 directories, it will be much harder to change the abstract distance of a memory_type?

Quoted 2 lines

I don't see any other reason we would want to expose memory type to userspace as of now.

Just like we expose the device tree to the user space via sysfs. Memory types are used to describe some hardware property directly. Users need these hardware information to manage their system.

Best Regards,
Huang, Ying

Quoted 6 lines

Quoted 4 lines
I want to get opinions from other people too.
Best Regards,
Huang, Ying
-aneesh

Aneesh Kumar K V

author

2022-09-05 06:15 UTC #23 lore raw

Quoted 169 lines · Ying

Quoted 167 lines · Aneesh Kumar K V
Quoted 165 lines · Ying
Quoted 143 lines · Aneesh Kumar K V
Quoted 133 lines · Ying
Quoted 131 lines · Aneesh Kumar K V
Quoted 129 lines · Ying
Quoted 113 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 54 lines · Ying
Quoted 52 lines · Aneesh Kumar K V
On 9/2/22 11:42 AM, Huang, Ying wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 11:10 AM, Huang, Ying wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 10:39 AM, Wei Xu wrote:
On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <[email protected]> wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
because we would want it to a susbsystem where all memory tiering related details can be found
including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one
subsystem (bus). If my understanding were correct, that breaks the
driver core convention.
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices.
memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows
details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have
almost totally different attributes (sysfs file). So, we should create
2 buses for them. Each has its own attribute group. "virtual" itself
isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate
them within the same subdirectory? I am still not clear why you are suggesting they need to be in different
sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
Quoted 8 lines
Quoted 2 lines
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should have mostly same attributes. This is my understanding of driver core convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
Quoted 8 lines
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.
Quoted 9 lines
Quoted 3 lines
I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.
I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types.
Quoted 7 lines
IMHO we can decide on that based on why we end up adding memory_type details to sysfs. If that is only for memory tier modification from userspace we can look at adding that in the memory tiering sysfs hierarchy.
Also since we have precedence of consolidating things within a sysfs hierarchy as explained in previous emails, I think we should keep "memory_tiering" as sysfs subsystem name? I hope we can get an agreement on that for now?
I prefer to separate memory_tier and memory_type, so the subsystem name should be "memory_tier". You prefer to consolidate memory_tier and memory_type, so the subsystem name should be "memory_tiering".
The main reason behind my idea is that memory_type isn't tied with memory tiering directly. It describes some hardware property. Even if we don't use memory tiering, we can still use that to classify the memory devices in the system.
Why do you want to consolidate them? To reduce one directory from sysfs?

Quoted 9 lines

Quoted 5 lines
So that it is much intuitive for user to got to memory_tiering sysfs hierarchy to change the memory tier levels. As I mentioned earlier the reason for consolidating things is to accommodate the possibility of supporting changing abstract distance of a memory type so that we can change the memory tier assignment of that specific
memory type.
If we put memory_tier and memory_type into 2 directories, it will be much harder to change the abstract distance of a memory_type?

I did explain I believe it is more intuitive to manage memory tier levels within memory tiering sysfs hierarchy. You seems to be ignoring my explanation in these emails.

Quoted 7 lines

Quoted 2 lines
I don't see any other reason we would want to expose memory type to userspace as of now.
Just like we expose the device tree to the user space via sysfs. Memory types are used to describe some hardware property directly. Users need these hardware information to manage their system.

Again explained in earlier emails already, I don't see a reason to duplicate attribute already present in /sys/devices/system/node/nodeY/access0/initiators/. Only reason we might end up adding memory type to sysfs is to manage memory tier levels. Hence the suggestion to consolidate things memory tiering directory.

-aneesh

Huang, Ying

2022-09-05 06:24 UTC #24 lore raw

Quoted 176 lines · Aneesh Kumar K V

Quoted 175 lines · Ying
Quoted 172 lines · Aneesh Kumar K V
Quoted 157 lines · Ying
Quoted 143 lines · Aneesh Kumar K V
Quoted 133 lines · Ying
Quoted 131 lines · Aneesh Kumar K V
Quoted 129 lines · Ying
Quoted 113 lines · Aneesh Kumar K V
Quoted 104 lines · Ying
Quoted 96 lines · Wei Xu
Quoted 72 lines · Aneesh Kumar K V
Quoted 62 lines · Ying
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 11:42 AM, Huang, Ying wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 11:10 AM, Huang, Ying wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/2/22 10:39 AM, Wei Xu wrote:
On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <[email protected]> wrote:
Aneesh Kumar K V <[email protected]> writes:
On 9/1/22 12:31 PM, Huang, Ying wrote:
"Aneesh Kumar K.V" <[email protected]> writes:
This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
related details can be found. All allocated memory tiers will be listed
there as /sys/devices/virtual/memory_tiering/memory_tierN/
The nodes which are part of a specific memory tier can be listed via
/sys/devices/virtual/memory_tiering/memory_tierN/nodes
I think "memory_tier" is a better subsystem/bus name than
memory_tiering. Because we have a set of memory_tierN devices inside.
"memory_tier" sounds more natural. I know this is subjective, just my
preference.
I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
because we would want it to a susbsystem where all memory tiering related details can be found
including memory type in the future. This is as per discussion
https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
I don't think that it's a good idea to mix 2 types of devices in one
subsystem (bus). If my understanding were correct, that breaks the
driver core convention.
All these are virtual devices .I am not sure i follow what you mean by 2 types of devices.
memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows
details of memory tiers and can possibly contain details of different memory types .
IMHO, memory_tier and memory_type are 2 kind of devices. They have
almost totally different attributes (sysfs file). So, we should create
2 buses for them. Each has its own attribute group. "virtual" itself
isn't a subsystem.
Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate
them within the same subdirectory? I am still not clear why you are suggesting they need to be in different
sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier.
/sys/devices/virtual/memory_tiering/memory_tierN
/sys/devices/virtual/memory_tiering/memory_typeN
I think we should add
/sys/devices/virtual/memory_tier/memory_tierN
/sys/devices/virtual/memory_type/memory_typeN
I am trying to find if there is a technical reason to do the same?
Quoted 4 lines
I don't think this is complex. Devices of same bus/subsystem should
have mostly same attributes. This is my understanding of driver core
convention.
Quoted 6 lines
I was not looking at this from code complexity point. Instead of having multiple directories with details w.r.t memory tiering, I was looking at consolidating the details within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices are consolidated within /sys/devics/virtual/).
-aneesh
Here is an example of /sys/bus/nd/devices (I know it is not under /sys/devices/virtual, but it can still serve as a reference):
ls -1 /sys/bus/nd/devices
namespace2.0
namespace3.0
ndbus0
nmem0
nmem1
region0
region1
region2
region3
Quoted 2 lines
So I think it is not unreasonable if we want to group memory tiering related interfaces within a single top directory.
Thanks for pointing this out. My original understanding of driver core isn't correct.
Quoted 7 lines
But I still think it's better to separate instead of mixing memory_tier and memory_type. Per my understanding, memory_type shows information (abstract distance, latency, bandwidth, etc.) of memory types (and nodes), it can be useful even without memory tiers. That is, memory types describes the physical characteristics, while memory tier reflects the policy.
Quoted 8 lines
The latency and bandwidth details are already exposed via
/sys/devices/system/node/nodeY/access0/initiators/
Documentation/admin-guide/mm/numaperf.rst
That is the interface that libraries like libmemkind will look at for finding details w.r.t latency/bandwidth
Yes. Only with that, it's still inconvenient to find out which nodes belong to same memory type (has same performance, same topology, managed by same driver, etc). So memory types can still provide useful information even without memory tiering.
Quoted 9 lines
Quoted 3 lines
I am not sure i quiet follow what to conclude from your reply. I used the subsystem name "memory_tiering" so that all memory tiering related information can be consolidated there. I guess you agreed to the above part that we can consolidated things like that.
I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types.
Quoted 7 lines
IMHO we can decide on that based on why we end up adding memory_type details to sysfs. If that is only for memory tier modification from userspace we can look at adding that in the memory tiering sysfs hierarchy.
Also since we have precedence of consolidating things within a sysfs hierarchy as explained in previous emails, I think we should keep "memory_tiering" as sysfs subsystem name? I hope we can get an agreement on that for now?
I prefer to separate memory_tier and memory_type, so the subsystem name should be "memory_tier". You prefer to consolidate memory_tier and memory_type, so the subsystem name should be "memory_tiering".
Quoted 8 lines
The main reason behind my idea is that memory_type isn't tied with memory tiering directly. It describes some hardware property. Even if we don't use memory tiering, we can still use that to classify the memory devices in the system.
Why do you want to consolidate them? To reduce one directory from sysfs?
So that it is much intuitive for user to got to memory_tiering sysfs hierarchy to change the memory tier levels. As I mentioned earlier the reason for consolidating things is to accommodate the possibility of supporting changing abstract distance of a memory type so that we can change the memory tier assignment of that specific
memory type.

Quoted 6 lines

Quoted 3 lines
If we put memory_tier and memory_type into 2 directories, it will be much harder to change the abstract distance of a memory_type?
I did explain I believe it is more intuitive to manage memory tier levels within memory tiering sysfs hierarchy. You seems to be ignoring my explanation in these emails.

I don't want to ignore your explanation. We just have different opinion. You think that it is more intuitive to put them in one hierarchy, while I think that it's more clear to separate them to reflect their difference.

Quoted 9 lines

Quoted 8 lines
Quoted 3 lines
I don't see any other reason we would want to expose memory type to userspace as of now.
Just like we expose the device tree to the user space via sysfs. Memory types are used to describe some hardware property directly. Users need these hardware information to manage their system.

Quoted 4 lines

Again explained in earlier emails already, I don't see a reason to duplicate attribute already present in /sys/devices/system/node/nodeY/access0/initiators/. Only reason we might end up adding memory type to sysfs is to manage memory tier levels. Hence the suggestion to consolidate things memory tiering directory.

I believe that we can provide useful information via memory_type in addition to memory tier related stuff. For example, we can know the memory type of each node, the driver to manage the memory type, name of memory type provided by driver, raw performance, etc.

Best Regards,
Huang, Ying