Documentation/admin-guide/sysctl/vm.rst adding the importance of NUMA-node count to documentation
Subsystems (3)
PATCH 1/1 Documentation/admin-guide/sysctl/vm.rst adding the importance of NUMA-node count to documentation
If any bits are set in node_reclaim_mode (tunable via /proc/sys/vm/zone_reclaim_mode) within get_pages_from_freelist(), then page allocations start getting early access to reclaim via the node_reclaim() code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:
Commit 9eeff2395e3cfd05c9b2e6 ("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."
While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in Documentation/ and isn't possible with a lone node. Imagine a situation where CONFIG_NUMA=y (the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user via vm.rst that the most bang for their buck is when multiple nodes exist seems helpful.
--- Documentation/admin-guide/sysctl/vm.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/admin-guide/sysctl/vm.rst +2 -1
@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the workload is partitioned such that each partition fits within a NUMA node and that accessing remote memory would cause a measurable performance reduction. The page allocator will take additional actions before -allocating off node pages. +allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode +makes the most sense for topologies consisting of multiple NUMA nodes. Allowing zone reclaim to write out pages stops processes that are writing large amounts of data from dirtying pages on other nodes. Zone
Signature · 1 line
2.34.1
Quoted 13 lines · Matthew Cassell
If any bits are set in
node_reclaim_mode(tunable via/proc/sys/vm/zone_reclaim_mode) withinget_pages_from_freelist(), then page allocations start getting early access to reclaim via thenode_reclaim()code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:Commit
9eeff2395e3cfd05c9b2e6("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."
Quoted 8 lines
While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in
Documentation/and isn't possible with a lone node. Imagine a situation whereCONFIG_NUMA=y(the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user viavm.rstthat the most bang for their buck is when multiple nodes exist seems helpful.
I agree that the documentation could be improved to better express the implications
and relevance of setting zone_reclaim_mode bits.
Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memory nodes."
Quoted 22 lines
Signed-off-by Matthew Cassell <[email protected]>--- Documentation/admin-guide/sysctl/vm.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Documentation/admin-guide/sysctl/vm.rst +2
-2@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the workload is partitioned such that each partition fits within a NUMA node and that accessing remote memory would cause a measurable performance reduction. The page allocator will take additional actions before -allocating off node pages. +allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode +makes the most sense for topologies consisting of multiple NUMA nodes. Allowing zone reclaim to write out pages stops processes that are writing large amounts of data from dirtying pages on other nodes. Zone --2.34.1
Thanks for the feedback. Here is a quick outline I came up with on your advice:
[...] (original content)
Keep in mind enabling bits in zone_reclaim_mode makes the most sense for topologies consisting of multiple NUMA nodes. In addition to vanilla zone_reclaim (clean and unmapped pages), there exist additional bits that expand which pages are eligible to be reclaimed and dictate scan_control policy during the reclaim process. The page allocator will attempt to reclaim memory locally in accordance with these bits before attempting to allocate on remote nodes.
Allow dirty pages to become candidates for memory reclaim::
echo 2 > /proc/sys/vm/zone_reclaim_mode
[...] (original content)
Allow mapped pages to become candidates for memory reclaim::
echo 4 > /proc/sys/vm/zone_reclaim_mode
[...] (original content)
I'm trying to balance between keeping the original content, being descriptive, and not going into encyclopedia-mode. My motivation was to stress the importance of NUMA-node count and describe the additional bits more per your advice. I added the echo snippets to better segue the aggressive options. Any thoughts on the above?
Quoted 58 lines · Vratislav Bendel
Quoted 13 lines · Matthew Cassell
If any bits are set in
node_reclaim_mode(tunable via/proc/sys/vm/zone_reclaim_mode) withinget_pages_from_freelist(), then page allocations start getting early access to reclaim via thenode_reclaim()code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:Commit
9eeff2395e3cfd05c9b2e6("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."Quoted 8 lines
While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in
Documentation/and isn't possible with a lone node. Imagine a situation whereCONFIG_NUMA=y(the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user viavm.rstthat the most bang for their buck is when multiple nodes exist seems helpful.I agree that the documentation could be improved to better express the implications
and relevance of settingzone_reclaim_modebits.Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memory nodes."Quoted 22 lines
Signed-off-by Matthew Cassell <[email protected]>--- Documentation/admin-guide/sysctl/vm.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Documentation/admin-guide/sysctl/vm.rst +2
-2@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the workload is partitioned such that each partition fits within a NUMA node and that accessing remote memory would cause a measurable performance reduction. The page allocator will take additional actions before -allocating off node pages. +allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode +makes the most sense for topologies consisting of multiple NUMA nodes. Allowing zone reclaim to write out pages stops processes that are writing large amounts of data from dirtying pages on other nodes. Zone --2.34.1
IMHO you went too encyclopedia-style this time. :)
My extra point was just to elaborate on what the "additional actions" means, but I would suggest keeping it as concise as possible. Also the bit values are already explained - no need to repeat that.
Quoted 107 lines · Matthew Cassell
Thanks for the feedback. Here is a quick outline I came up with on your advice:
[...] (original content)
Keep in mind enabling bits in
zone_reclaim_modemakes the most sense for topologies consisting of multiple NUMA nodes. In addition to vanillazone_reclaim(clean and unmapped pages), there exist additional bits that expand which pages are eligible to be reclaimed and dictatescan_controlpolicy during the reclaim process. The page allocator will attempt to reclaim
memory locally in accordance with these bits before attempting to allocate on remote nodes.Allow dirty pages to become candidates for memory reclaim::
echo 2 >
/proc/sys/vm/zone_reclaim_mode[...] (original content)
Allow mapped pages to become candidates for memory reclaim::
echo 4 >
/proc/sys/vm/zone_reclaim_mode[...] (original content)
I'm trying to balance between keeping the original content, being descriptive,
and not going into encyclopedia-mode. My motivation was to stress the importance
of NUMA-node count and describe the additional bits more per your advice. I added the echo snippets to better segue the aggressive options. Any thoughts
on the above?Quoted 2 lines · Vratislav Bendel
On Fri, Apr 5, 2024 at 6:49 PM Matthew Cassell <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
If any bits are set in
node_reclaim_mode(tunable via/proc/sys/vm/zone_reclaim_mode) withinget_pages_from_freelist(), then page allocations start getting early access to reclaim via thenode_reclaim()code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. Theabove
Quoted 10 lines
Quoted 10 lines
is mentioned in:
Commit
9eeff2395e3cfd05c9b2e6("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in
Documentation/and isn't possible with a lone node.Imagine a
Quoted 3 lines
Quoted 3 lines
situation where
CONFIG_NUMA=y(the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user viavm.rstthat
Quoted 1 line
Quoted 1 line
the most bang for their buck is when multiple nodes exist seems
helpful.
Quoted 6 lines
Quoted 1 line
I agree that the documentation could be improved to better express the implications
and relevance of settingzone_reclaim_modebits.Quoted 6 lines
Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memorynodes."
Quoted 7 lines
Review
Quoted 7 lines
Signed-off-by Matthew Cassell <[email protected]>--- Documentation/admin-guide/sysctl/vm.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/Documentation/admin-guide/sysctl/vm.rstQuoted 4 lines
Documentation/admin-guide/sysctl/vm.rst Consider enabling one or more zone_reclaim mode
Quoted 4 lines
index
c59889de122b..10270548af2a100644Documentation/admin-guide/sysctl/vm.rst +0
-0@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim modebits if it's known that the
Quoted 1 line
Documentation/admin-guide/sysctl/vm.rst Consider enabling one or more zone_reclaim mode · line 1031
Quoted 1 line
Documentation/admin-guide/sysctl/vm.rst +0
-0workload is partitioned such that each partition fits within a NUMAnode
Quoted 4 lines
Documentation/admin-guide/sysctl/vm.rst Consider enabling one or more zone_reclaim mode bits if it's known that the
Quoted 4 lines
Documentation/admin-guide/sysctl/vm.rst +1
-1and that accessing remote memory would cause a measurable performance reduction. The page allocator will take additional actions before -allocating off node pages. +allocating off node pages. Keep in mind enabling bits in
zone_reclaim_modeQuoted 8 lines
Quoted 7 lines
Documentation/admin-guide/sysctl/vm.rst +1
-1+makes the most sense for topologies consisting of multiple NUMA nodes. Allowing zone reclaim to write out pages stops processes that are writing large amounts of data from dirtying pages on other nodes. Zone --2.34.1
(plain text resend)
IMHO you went too encyclopedia-style this time. :)
My extra point was just to elaborate on what the "additional actions" means, but I would suggest keeping it as concise as possible.
Also the bit values are already explained - no need to repeat that.
Quoted 92 lines · Matthew Cassell
Thanks for the feedback. Here is a quick outline I came up with on your advice:
[...] (original content)
Keep in mind enabling bits in
zone_reclaim_modemakes the most sense for topologies consisting of multiple NUMA nodes. In addition to vanillazone_reclaim(clean and unmapped pages), there exist additional bits that expand which pages are eligible to be reclaimed and dictatescan_controlpolicy during the reclaim process. The page allocator will attempt to reclaim memory locally in accordance with these bits before attempting to allocate on remote nodes.Allow dirty pages to become candidates for memory reclaim::
echo 2 >
/proc/sys/vm/zone_reclaim_mode[...] (original content)
Allow mapped pages to become candidates for memory reclaim::
echo 4 >
/proc/sys/vm/zone_reclaim_mode[...] (original content)
I'm trying to balance between keeping the original content, being descriptive, and not going into encyclopedia-mode. My motivation was to stress the importance of NUMA-node count and describe the additional bits more per your advice. I added the echo snippets to better segue the aggressive options. Any thoughts on the above?
Quoted 58 lines · Vratislav Bendel
Quoted 13 lines · Matthew Cassell
If any bits are set in
node_reclaim_mode(tunable via/proc/sys/vm/zone_reclaim_mode) withinget_pages_from_freelist(), then page allocations start getting early access to reclaim via thenode_reclaim()code path when memory pressure increases. This behavior provides the most optimization for multiple NUMA node machines. The above is mentioned in:Commit
9eeff2395e3cfd05c9b2e6("[PATCH] Zone reclaim: Reclaim logic") states "Zone reclaim is of particular importance for NUMA machines. It can be more beneficial to reclaim a page than taking the performance penalties that come with allocating a page on a REMOTE zone."Quoted 8 lines
While the pros/cons of staying on node versus allocating remotely are mentioned in commit histories and mailing lists. It isn't specifically mentioned in
Documentation/and isn't possible with a lone node. Imagine a situation whereCONFIG_NUMA=y(the default on most major distributions) and only a single NUMA node exists. The latter is an oxymoron (single-node == uniform memory access). Informing the user viavm.rstthat the most bang for their buck is when multiple nodes exist seems helpful.I agree that the documentation could be improved to better express the implications
and relevance of settingzone_reclaim_modebits.Though I would suggest to go a step further and also elaborate on those "additional actions",
for example something like:
"The page allocator will attempt to reclaim memory within the zone, depending on the bits set,
before looking for free pages in other zones, namely on remote memory nodes."Quoted 22 lines
Signed-off-by Matthew Cassell <[email protected]>--- Documentation/admin-guide/sysctl/vm.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Documentation/admin-guide/sysctl/vm.rst +2
-2@@ -1031,7 +1031,8 @@ Consider enabling one or more zone_reclaim mode bits if it's known that the workload is partitioned such that each partition fits within a NUMA node and that accessing remote memory would cause a measurable performance reduction. The page allocator will take additional actions before -allocating off node pages. +allocating off node pages. Keep in mind enabling bits in zone_reclaim_mode +makes the most sense for topologies consisting of multiple NUMA nodes. Allowing zone reclaim to write out pages stops processes that are writing large amounts of data from dirtying pages on other nodes. Zone --2.34.1