Skip to contentSkip to conversation

Documentation/admin-guide/sysctl/vm.rst adding the importance of NUMA-node count to documentation

PATCH In review Matthew Cassell 2 participants 5 messages Apr 5, 2024 – Apr 27, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Documentation/admin-guide/sysctl/vm.rst adding the importance of NUMA-node count to documentation

If any bits are set in node_reclaim_mode (tunable via /proc/sys/vm/zone_reclaim_mode) within get_pages_from_freelist(), then page allocations start getting early access to reclaim via the node_reclaim() code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:

Commit 9eeff2395e3cfd05c9b2e6 ("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."

While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in Documentation/ and isn't possible with a lone node. Imagine a situation where CONFIG_NUMA=y (the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user via vm.rst that the most bang for their buck is when multiple nodes exist seems helpful.

---
 Documentation/admin-guide/sysctl/vm.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/admin-guide/sysctl/vm.rst +2 -1
index c59889de122b..10270548af2a 100644
@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the
 workload is partitioned such that each partition fits within a NUMA node
 and that accessing remote memory would cause a measurable performance
 reduction.  The page allocator will take additional actions before
-allocating off node pages.
+allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode
+makes the most sense for topologies consisting of multiple NUMA nodes.
 
 Allowing zone reclaim to write out pages stops processes that are
 writing large amounts of data from dirtying pages on other nodes. Zone
Signature · 1 line
2.34.1
#1 lore raw
Quoted 13 lines · Matthew Cassell

If any bits are set in node_reclaim_mode (tunable via /proc/sys/vm/zone_reclaim_mode) within get_pages_from_freelist(), then page allocations start getting early access to reclaim via the node_reclaim() code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:

Commit 9eeff2395e3cfd05c9b2e6 ("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."

Quoted 8 lines

While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in Documentation/ and isn't possible with a lone node. Imagine a situation where CONFIG_NUMA=y (the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user via vm.rst that the most bang for their buck is when multiple nodes exist seems helpful.

I agree that the documentation could be improved to better express the implications
and relevance of setting zone_reclaim_mode bits.

Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memory nodes."

Quoted 22 lines
---
 Documentation/admin-guide/sysctl/vm.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/admin-guide/sysctl/vm.rst +2 -2
index c59889de122b..10270548af2a 100644
@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the
 workload is partitioned such that each partition fits within a NUMA node
 and that accessing remote memory would cause a measurable performance
 reduction.  The page allocator will take additional actions before
-allocating off node pages.
+allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode
+makes the most sense for topologies consisting of multiple NUMA nodes.
 
 Allowing zone reclaim to write out pages stops processes that are
 writing large amounts of data from dirtying pages on other nodes. Zone
--

2.34.1

#2 lore raw

Thanks for the feedback. Here is a quick outline I came up with on your advice:

[...] (original content)

Keep in mind enabling bits in zone_reclaim_mode makes the most sense for topologies consisting of multiple NUMA nodes. In addition to vanilla zone_reclaim (clean and unmapped pages), there exist additional bits that expand which pages are eligible to be reclaimed and dictate scan_control policy during the reclaim process. The page allocator will attempt to reclaim memory locally in accordance with these bits before attempting to allocate on remote nodes.

Allow dirty pages to become candidates for memory reclaim::

echo 2 > /proc/sys/vm/zone_reclaim_mode

[...] (original content)

Allow mapped pages to become candidates for memory reclaim::

echo 4 > /proc/sys/vm/zone_reclaim_mode

[...] (original content)

I'm trying to balance between keeping the original content, being descriptive, and not going into encyclopedia-mode. My motivation was to stress the importance of NUMA-node count and describe the additional bits more per your advice. I added the echo snippets to better segue the aggressive options. Any thoughts on the above?

Quoted 58 lines · Vratislav Bendel
Quoted 13 lines · Matthew Cassell

If any bits are set in node_reclaim_mode (tunable via /proc/sys/vm/zone_reclaim_mode) within get_pages_from_freelist(), then page allocations start getting early access to reclaim via the node_reclaim() code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:

Commit 9eeff2395e3cfd05c9b2e6 ("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."

Quoted 8 lines

While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in Documentation/ and isn't possible with a lone node. Imagine a situation where CONFIG_NUMA=y (the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user via vm.rst that the most bang for their buck is when multiple nodes exist seems helpful.

I agree that the documentation could be improved to better express the implications
and relevance of setting zone_reclaim_mode bits.

Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memory nodes."

Quoted 22 lines
---
 Documentation/admin-guide/sysctl/vm.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/admin-guide/sysctl/vm.rst +2 -2
index c59889de122b..10270548af2a 100644
@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the
 workload is partitioned such that each partition fits within a NUMA node
 and that accessing remote memory would cause a measurable performance
 reduction.  The page allocator will take additional actions before
-allocating off node pages.
+allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode
+makes the most sense for topologies consisting of multiple NUMA nodes.
 
 Allowing zone reclaim to write out pages stops processes that are
 writing large amounts of data from dirtying pages on other nodes. Zone
--

2.34.1

#3 lore raw

IMHO you went too encyclopedia-style this time. :)
My extra point was just to elaborate on what the "additional actions" means, but I would suggest keeping it as concise as possible. Also the bit values are already explained - no need to repeat that.

Quoted 107 lines · Matthew Cassell

Thanks for the feedback. Here is a quick outline I came up with on your advice:

[...] (original content)

Keep in mind enabling bits in zone_reclaim_mode makes the most sense for topologies consisting of multiple NUMA nodes. In addition to vanilla zone_reclaim (clean and unmapped pages), there exist additional bits that expand which pages are eligible to be reclaimed and dictate scan_control policy during the reclaim process. The page allocator will attempt to reclaim
memory locally in accordance with these bits before attempting to allocate on remote nodes.

Allow dirty pages to become candidates for memory reclaim::

echo 2 > /proc/sys/vm/zone_reclaim_mode

[...] (original content)

Allow mapped pages to become candidates for memory reclaim::

echo 4 > /proc/sys/vm/zone_reclaim_mode

[...] (original content)

I'm trying to balance between keeping the original content, being descriptive,
and not going into encyclopedia-mode. My motivation was to stress the importance
of NUMA-node count and describe the additional bits more per your advice. I added the echo snippets to better segue the aggressive options. Any thoughts
on the above?

Quoted 2 lines · Vratislav Bendel

On Fri, Apr 5, 2024 at 6:49 PM Matthew Cassell <[email protected]>

Quoted 6 lines · wrote:
Quoted 6 lines

If any bits are set in node_reclaim_mode (tunable via /proc/sys/vm/zone_reclaim_mode) within get_pages_from_freelist(), then page allocations start getting early access to reclaim via the node_reclaim() code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The

above

Quoted 10 lines
Quoted 10 lines

is mentioned in:

Commit 9eeff2395e3cfd05c9b2e6 ("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."

While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in Documentation/ and isn't possible with a lone node.

Imagine a

Quoted 3 lines
Quoted 3 lines

situation where CONFIG_NUMA=y (the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user via vm.rst

that

Quoted 1 line
Quoted 1 line

the most bang for their buck is when multiple nodes exist seems

helpful.

Quoted 6 lines
Quoted 1 line

I agree that the documentation could be improved to better express the implications
and relevance of setting zone_reclaim_mode bits.

Quoted 6 lines

Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memory

nodes."

Quoted 7 lines

Review

Quoted 7 lines
---
 Documentation/admin-guide/sysctl/vm.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/Documentation/admin-guide/sysctl/vm.rst

Quoted 4 lines

Documentation/admin-guide/sysctl/vm.rst Consider enabling one or more zone_reclaim mode

Quoted 4 lines

index c59889de122b..10270548af2a 100644

Documentation/admin-guide/sysctl/vm.rst +0 -0
@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode

bits if it's known that the

Quoted 1 line

Documentation/admin-guide/sysctl/vm.rst Consider enabling one or more zone_reclaim mode · line 1031

Quoted 1 line
Documentation/admin-guide/sysctl/vm.rst +0 -0
 workload is partitioned such that each partition fits within a NUMA

node

Quoted 4 lines

Documentation/admin-guide/sysctl/vm.rst Consider enabling one or more zone_reclaim mode bits if it's known that the

Quoted 4 lines
Documentation/admin-guide/sysctl/vm.rst +1 -1
 and that accessing remote memory would cause a measurable performance
 reduction.  The page allocator will take additional actions before
-allocating off node pages.
+allocating off node pages. Keep in mind enabling bits in

zone_reclaim_mode

Quoted 8 lines
Quoted 7 lines
Documentation/admin-guide/sysctl/vm.rst +1 -1
+makes the most sense for topologies consisting of multiple NUMA nodes.
 
 Allowing zone reclaim to write out pages stops processes that are
 writing large amounts of data from dirtying pages on other nodes. Zone
--

2.34.1

#4 lore raw
In reply to Matthew Cassell

(plain text resend)

IMHO you went too encyclopedia-style this time. :)
My extra point was just to elaborate on what the "additional actions" means, but I would suggest keeping it as concise as possible.
Also the bit values are already explained - no need to repeat that.

Quoted 92 lines · Matthew Cassell

Thanks for the feedback. Here is a quick outline I came up with on your advice:

[...] (original content)

Keep in mind enabling bits in zone_reclaim_mode makes the most sense for topologies consisting of multiple NUMA nodes. In addition to vanilla zone_reclaim (clean and unmapped pages), there exist additional bits that expand which pages are eligible to be reclaimed and dictate scan_control policy during the reclaim process. The page allocator will attempt to reclaim memory locally in accordance with these bits before attempting to allocate on remote nodes.

Allow dirty pages to become candidates for memory reclaim::

echo 2 > /proc/sys/vm/zone_reclaim_mode

[...] (original content)

Allow mapped pages to become candidates for memory reclaim::

echo 4 > /proc/sys/vm/zone_reclaim_mode

[...] (original content)

I'm trying to balance between keeping the original content, being descriptive, and not going into encyclopedia-mode. My motivation was to stress the importance of NUMA-node count and describe the additional bits more per your advice. I added the echo snippets to better segue the aggressive options. Any thoughts on the above?

Quoted 58 lines · Vratislav Bendel
Quoted 13 lines · Matthew Cassell

If any bits are set in node_reclaim_mode (tunable via /proc/sys/vm/zone_reclaim_mode) within get_pages_from_freelist(), then page allocations start getting early access to reclaim via the node_reclaim() code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:

Commit 9eeff2395e3cfd05c9b2e6 ("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."

Quoted 8 lines

While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in Documentation/ and isn't possible with a lone node. Imagine a situation where CONFIG_NUMA=y (the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user via vm.rst that the most bang for their buck is when multiple nodes exist seems helpful.

I agree that the documentation could be improved to better express the implications
and relevance of setting zone_reclaim_mode bits.

Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memory nodes."

Quoted 22 lines
---
 Documentation/admin-guide/sysctl/vm.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/admin-guide/sysctl/vm.rst +2 -2
index c59889de122b..10270548af2a 100644
@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the
 workload is partitioned such that each partition fits within a NUMA node
 and that accessing remote memory would cause a measurable performance
 reduction.  The page allocator will take additional actions before
-allocating off node pages.
+allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode
+makes the most sense for topologies consisting of multiple NUMA nodes.
 
 Allowing zone reclaim to write out pages stops processes that are
 writing large amounts of data from dirtying pages on other nodes. Zone
--

2.34.1