mm/ksm: clean up scan_get_next_rmap_item() initialization
Xu Xin (vivo)
reviewercover mm/ksm: clean up scan_get_next_rmap_item() initialization
Currently, scan_get_next_rmap_item() has grown quite large and complex. When introducing new features, such as the only-merge-zero-page mode we are currently working on, we don't want to add more complexity to it and don't want to touch rmap_item structures to avoid unnecessary overhead, so we need to extract some common logic.
With these changes, the entry of scan_get_next_rmap_item() becomes much cleaner and easier to follow, instead of being cluttered with unrelated boilerplate such as stale stable_node pruning and cycle initialization.
Let's start by cleaning up these two parts at the beginning (where I think abstraction is necessary), and we can continue cleaning up further if needed later.
Xu Xin (vivo) (2):
mm/ksm: extract helper functions for scan cycle cleanup mm/ksm: add initialization helpers for scan_get_next_rmap_item()
mm/ksm.c | 126 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 79 insertions(+), 47 deletions(-)
Signature · 1 line
2.34.1
Xu Xin (vivo)
reviewerPATCH 1/2 mm/ksm: extract helper functions for scan cycle cleanup
Since scan_get_next_rmap_item() is getting too large, extract two new helper functions, ksm_prune_migrate_nodes() and ksm_reset_unstable_trees(), to reduce indentation levels, isolate local variables, and improve the readability of the main scan loop.
No functional change intended.
--- mm/ksm.c | 56 +++++++++++++++++++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 23 deletions(-)
mm/ksm.c +33 -23
@@ -2609,6 +2609,37 @@ static struct mm_walk_ops ksm_next_page_ops = { .walk_lock = PGWALK_RDLOCK, }; +static void ksm_prune_migrate_nodes (void ) +{ +struct ksm_stable_node *stable_node, *next; +struct folio *folio; + +if (ksm_merge_across_nodes) +return ; + +/* +* Whereas stale stable_nodes on the stable_tree itself +* get pruned in the regular course of stable_tree_search(), +* those moved out to the migrate_nodes list can accumulate: +* so prune them once before each full scan. +*/ +list_for_each_entry_safe (stable_node, next, &migrate_nodes, list) { + folio =ksm_get_folio (stable_node,KSM_GET_FOLIO_NOLOCK ); + +if (folio) +folio_put (folio); +cond_resched (); + } +} + +static void ksm_reset_unstable_trees (void ) +{ +int nid; + +for (nid =0 ; nid < ksm_nr_node_ids; nid++) + root_unstable_tree[nid] =RB_ROOT ; +} +static struct ksm_rmap_item *scan_get_next_rmap_item (struct page **page) {struct mm_struct *mm;
@@ -2617,7 +2648,6 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)struct vm_area_struct *vma;struct ksm_rmap_item *rmap_item;struct vma_iterator vmi; -int nid;if (list_empty (&ksm_mm_head.slot.mm_node))return NULL ;
@@ -2639,28 +2669,8 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)*/ lru_add_drain_all (); -/* -* Whereas stale stable_nodes on the stable_tree itself -* get pruned in the regular course of stable_tree_search(), -* those moved out to the migrate_nodes list can accumulate: -* so prune them once before each full scan. -*/ -if (!ksm_merge_across_nodes) { -struct ksm_stable_node *stable_node, *next; -struct folio *folio; - -list_for_each_entry_safe (stable_node, next, - &migrate_nodes, list) { - folio =ksm_get_folio (stable_node, -KSM_GET_FOLIO_NOLOCK ); -if (folio) -folio_put (folio); -cond_resched (); - } - } - -for (nid =0 ; nid < ksm_nr_node_ids; nid++) - root_unstable_tree[nid] =RB_ROOT ; +ksm_prune_migrate_nodes (); +ksm_reset_unstable_trees ();spin_lock (&ksm_mmlist_lock); slot = list_entry(mm_slot->slot.mm_node.next,
Signature · 1 line
2.34.1
Re: [PATCH 1/2] mm/ksm: extract helper functions for scan cycle cleanup
Quoted 8 lines · Xu Xin (vivo)
Since
scan_get_next_rmap_item()is getting too large, extract two new helper functions,ksm_prune_migrate_nodes()andksm_reset_unstable_trees(), to reduce indentation levels, isolate local variables, and improve the readability of the main scan loop.No functional change intended.
Signed-off-by Xu Xin (vivo) <[email protected]>
Thanks for doing this! LGTM so:
One small nit below, but it's not a big deal :)
Quoted 8 lines
--- mm/ksm.c | 56 +++++++++++++++++++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 23 deletions(-)mm/ksm.c +0
-0
[ ... ]
mm/ksm.c static struct mm_walk_ops ksm_next_page_ops = { · line 2641
Quoted 7 lines
mm/ksm.c +7
-0+static void ksm_reset_unstable_trees (void ) +{ +int nid; + +for (nid =0 ; nid < ksm_nr_node_ids; nid++) + root_unstable_tree[nid] =RB_ROOT ; +}
Recently I learned (and PeterZ reminded me :) that it's actually valid to declare the var in the for-block in linux kernel C.
So this can be:
static void ksm_reset_unstable_trees(void)
{
for (int nid = 0; nid < ksm_nr_node_ids; nid++)
root_unstable_tree[nid] = RB_ROOT;
}:)
--
Cheers, Lorenzo
Re: [PATCH 1/2] mm/ksm: extract helper functions for scan cycle cleanup
In reply to Xu Xin (vivo)Quoted 9 lines · Xu Xin (vivo)
Since
scan_get_next_rmap_item()is getting too large, extract two new helper functions,ksm_prune_migrate_nodes()andksm_reset_unstable_trees(), to reduce indentation levels, isolate local variables, and improve the readability of the main scan loop.No functional change intended.
Signed-off-by Xu Xin (vivo) <[email protected]>---
Signature · 3 lines
Cheers, David
Xu Xin (vivo)
reviewerPATCH 2/2 mm/ksm: add initialization helpers for scan_get_next_rmap_item()
In reply to Xu Xin (vivo)Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first mm_slot into scan_begin_new_cycle() and scan_get_first_mm().
This keeps the exact execution order intact while making scan_get_next_rmap_item() cleaner and easier to follow.
No functional change intended.
--- mm/ksm.c | 70 +++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 24 deletions(-)
mm/ksm.c +46 -24
@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void) root_unstable_tree[nid] =RB_ROOT ; } +static bool scan_begin_new_cycle (void ) +{ +/* Check if the cursor points to the head. */ +if (ksm_scan.mm_slot != &ksm_mm_head) +return false; + +/* +* A number of pages can hang around indefinitely in per-cpu +* LRU cache, raised page count preventing write_protect_page +* from merging them. Though it doesn't really matter much, +* it is puzzling to see some stuck in pages_volatile until +* other activity jostles them out, and they also prevented +* LTP's KSM test from succeeding deterministically; so drain +* them here (here rather than on entry to ksm_do_scan(), +* so we don't IPI too often when pages_to_scan is set low). +*/ +lru_add_drain_all (); +return true; +} + +static struct ksm_mm_slot *scan_get_first_mm (void ) +{ +struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot; +struct mm_slot *slot; + +spin_lock (&ksm_mmlist_lock); + + slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node); + mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot); + ksm_scan.mm_slot = mm_slot; + +spin_unlock (&ksm_mmlist_lock); + +/* +* Although we tested list_empty() above, a racing __ksm_exit +* of the last mm on the list may have removed it since then. +*/ +if (mm_slot == &ksm_mm_head) +return NULL ; + +return mm_slot; +} +static struct ksm_rmap_item *scan_get_next_rmap_item (struct page **page) {struct mm_struct *mm;
@@ -2653,36 +2696,15 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)return NULL ; mm_slot = ksm_scan.mm_slot; -if (mm_slot == &ksm_mm_head) { +if (scan_begin_new_cycle ()) {advisor_start_scan ();trace_ksm_start_scan (ksm_scan.seqnr, ksm_rmap_items); -/* -* A number of pages can hang around indefinitely in per-cpu -* LRU cache, raised page count preventing write_protect_page -* from merging them. Though it doesn't really matter much, -* it is puzzling to see some stuck in pages_volatile until -* other activity jostles them out, and they also prevented -* LTP's KSM test from succeeding deterministically; so drain -* them here (here rather than on entry to ksm_do_scan(), -* so we don't IPI too often when pages_to_scan is set low). -*/ -lru_add_drain_all (); -ksm_prune_migrate_nodes ();ksm_reset_unstable_trees (); -spin_lock (&ksm_mmlist_lock); - slot =list_entry (mm_slot->slot.mm_node.next, - struct mm_slot, mm_node); - mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot); - ksm_scan.mm_slot = mm_slot; -spin_unlock (&ksm_mmlist_lock); -/* -* Although we tested list_empty() above, a racing __ksm_exit -* of the last mm on the list may have removed it since then. -*/ -if (mm_slot == &ksm_mm_head) + mm_slot =scan_get_first_mm (); +if (!mm_slot)return NULL ; next_mm: ksm_scan.address =0 ;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/ksm: scan_begin_new_cycle() reorders lru_add_drain_all() at scan start --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/ksm: scan_begin_new_cycle() reorders lru_add_drain_all() at scan start
--
commit 82208b052cdf6d037a169e81cb68de5a2ad7f1fc
Author: Xu Xin (vivo) <[email protected]>
mm/ksm: add initialization helpers for scan_get_next_rmap_item()
This commit extracts the new scan cycle check and the first mm_slot lookup from scan_get_next_rmap_item() into scan_begin_new_cycle() and scan_get_first_mm().
[Severity: Low]
In mm/ksm.c, does moving lru_add_drain_all() into scan_begin_new_cycle() change the execution order at the start of a scan cycle in scan_get_next_rmap_item()?
Previously, when ksm_scan.mm_slot equaled &ksm_mm_head, advisor_start_scan() and trace_ksm_start_scan() ran before lru_add_drain_all(). Calling lru_add_drain_all() inside scan_begin_new_cycle() in the if condition runs it before both calls, contrary to the commit message's statement that the exact execution order is kept intact.
This reverses the relative order of the mm_lru_add_drain_all and ksm_start_scan tracepoints. It also records advisor_ctx.start_scan via ktime_get() in advisor_start_scan() only after lru_add_drain_all() finishes draining per-CPU LRU caches, whereas scan_time_advisor() still includes the ksmd CPU runtime spent inside lru_add_drain_all() via task_sched_runtime(current).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()
In reply to Xu Xin (vivo)Quoted 10 lines · Xu Xin (vivo)
Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first
mm_slotintoscan_begin_new_cycle()andscan_get_first_mm().This keeps the exact execution order intact while making
scan_get_next_rmap_item()cleaner and easier to follow.No functional change intended.
Signed-off-by Xu Xin (vivo) <[email protected]>
Again, thanks for doing this! LGTM and:
Quoted 19 lines
--- mm/ksm.c | 70 +++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 24 deletions(-)mm/ksm.c +7
-0@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void) root_unstable_tree[nid] =RB_ROOT ; } +static bool scan_begin_new_cycle (void ) +{ +/* Check if the cursor points to the head. */ +if (ksm_scan.mm_slot != &ksm_mm_head) +return false; + +/*
mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2659
Quoted 10 lines
mm/ksm.c +10
-0+* A number of pages can hang around indefinitely in per-cpu +* LRU cache, raised page count preventing write_protect_page +* from merging them. Though it doesn't really matter much, +* it is puzzling to see some stuck in pages_volatile until +* other activity jostles them out, and they also prevented +* LTP's KSM test from succeeding deterministically; so drain +* them here (here rather than on entry to ksm_do_scan(), +* so we don't IPI too often when pages_to_scan is set low). +*/ +lru_add_drain_all ();
The sashiko finding is silly, this reorders it after advisor_start_scan() which calls... ktime_get(), yeah it's zero delta :)
Quoted 5 lines
mm/ksm.c +5
-0+return true; +} + +static struct ksm_mm_slot *scan_get_first_mm (void ) +{
mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2674
Quoted 10 lines
mm/ksm.c +10
-0+struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot; +struct mm_slot *slot; + +spin_lock (&ksm_mmlist_lock); + + slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node); + mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot); + ksm_scan.mm_slot = mm_slot; + +spin_unlock (&ksm_mmlist_lock);
NIT: optional, but this could be:
scoped_guard(mutex, &ksm_mmlist_lock) {
slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
ksm_scan.mm_slot = mm_slot;
}Quoted 55 lines
mm/ksm.c +14
-25+ +/* +* Although we tested list_empty() above, a racing __ksm_exit +* of the last mm on the list may have removed it since then. +*/ +if (mm_slot == &ksm_mm_head) +return NULL ; + +return mm_slot; +} +static struct ksm_rmap_item *scan_get_next_rmap_item (struct page **page) {struct mm_struct *mm;@@ -2653,36 +2696,15 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)return NULL ; mm_slot = ksm_scan.mm_slot; -if (mm_slot == &ksm_mm_head) { +if (scan_begin_new_cycle ()) {advisor_start_scan ();trace_ksm_start_scan (ksm_scan.seqnr, ksm_rmap_items); -/* -* A number of pages can hang around indefinitely in per-cpu -* LRU cache, raised page count preventing write_protect_page -* from merging them. Though it doesn't really matter much, -* it is puzzling to see some stuck in pages_volatile until -* other activity jostles them out, and they also prevented -* LTP's KSM test from succeeding deterministically; so drain -* them here (here rather than on entry to ksm_do_scan(), -* so we don't IPI too often when pages_to_scan is set low). -*/ -lru_add_drain_all (); -ksm_prune_migrate_nodes ();ksm_reset_unstable_trees (); -spin_lock (&ksm_mmlist_lock); - slot =list_entry (mm_slot->slot.mm_node.next, - struct mm_slot, mm_node); - mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot); - ksm_scan.mm_slot = mm_slot; -spin_unlock (&ksm_mmlist_lock); -/* -* Although we tested list_empty() above, a racing __ksm_exit -* of the last mm on the list may have removed it since then. -*/ -if (mm_slot == &ksm_mm_head) + mm_slot =scan_get_first_mm (); +if (!mm_slot)return NULL ; next_mm: ksm_scan.address =0 ; --
Quoted 2 lines
2.34.1
--
Cheers, Lorenzo
Xu Xin (vivo)
reviewerRe: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()
Quoted 8 lines · Xu Xin (vivo)
Quoted 7 lines
Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first
mm_slotintoscan_begin_new_cycle()andscan_get_first_mm().This keeps the exact execution order intact while making
scan_get_next_rmap_item()cleaner and easier to follow.
Quoted 7 lines
Quoted 3 lines
No functional change intended.
Signed-off-by Xu Xin (vivo) <[email protected]>Again, thanks for doing this! LGTM and:
Reviewed-by Lorenzo Stoakes (ARM) <[email protected]>
Thanks.
Quoted 16 lines
mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2674
Quoted 13 lines
mm/ksm.c +12
-0+static struct ksm_mm_slot *scan_get_first_mm (void ) +{ +struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot; +struct mm_slot *slot; + +spin_lock (&ksm_mmlist_lock); + + slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node); + mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot); + ksm_scan.mm_slot = mm_slot; + +spin_unlock (&ksm_mmlist_lock);NIT: optional, but this could be:
Quoted 7 lines
scoped_guard(mutex, &ksm_mmlist_lock) {
slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
ksm_scan.mm_slot = mm_slot;
}Good idea, I will switch to scoped_guard.
David Hildenbrand (Arm)
maintainerRe: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()
In reply to Xu Xin (vivo)Quoted 54 lines · Xu Xin (vivo)
Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first
mm_slotintoscan_begin_new_cycle()andscan_get_first_mm().This keeps the exact execution order intact while making
scan_get_next_rmap_item()cleaner and easier to follow.No functional change intended.
Signed-off-by Xu Xin (vivo) <[email protected]>--- mm/ksm.c | 70 +++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 24 deletions(-)mm/ksm.c +32
-0@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void) root_unstable_tree[nid] =RB_ROOT ; } +static bool scan_begin_new_cycle (void ) +{ +/* Check if the cursor points to the head. */ +if (ksm_scan.mm_slot != &ksm_mm_head) +return false; + +/* +* A number of pages can hang around indefinitely in per-cpu +* LRU cache, raised page count preventing write_protect_page +* from merging them. Though it doesn't really matter much, +* it is puzzling to see some stuck in pages_volatile until +* other activity jostles them out, and they also prevented +* LTP's KSM test from succeeding deterministically; so drain +* them here (here rather than on entry to ksm_do_scan(), +* so we don't IPI too often when pages_to_scan is set low). +*/ +lru_add_drain_all (); +return true; +} + +static struct ksm_mm_slot *scan_get_first_mm (void ) +{ +struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot; +struct mm_slot *slot; + +spin_lock (&ksm_mmlist_lock); + + slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node); + mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot); + ksm_scan.mm_slot = mm_slot; + +spin_unlock (&ksm_mmlist_lock);
mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2684
Quoted 10 lines
mm/ksm.c +10
-0+ +/* +* Although we tested list_empty() above, a racing __ksm_exit +* of the last mm on the list may have removed it since then. +*/ +if (mm_slot == &ksm_mm_head) +return NULL ; + +return mm_slot; +}
I like this function cleanup here, but not sure about the other one.
Can we just leave the lru_add_drain_all() where it is, or what's the reason for that change?
Note that that might soon get removed either way, so I wouldn't worry about it.
Then, I would rename the function into something like
"scan_should_start_new_cycle"
And keep it simple.
Signature · 3 lines
Cheers, David
Xu Xin (vivo)
reviewerRe: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()
Quoted 23 lines
Quoted 23 lines
mm/ksm.c +15
-0@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void) root_unstable_tree[nid] =RB_ROOT ; } +static bool scan_begin_new_cycle (void ) +{ +/* Check if the cursor points to the head. */ +if (ksm_scan.mm_slot != &ksm_mm_head) +return false; + +/* +* A number of pages can hang around indefinitely in per-cpu +* LRU cache, raised page count preventing write_protect_page +* from merging them. Though it doesn't really matter much, +* it is puzzling to see some stuck in pages_volatile until +* other activity jostles them out, and they also prevented +* LTP's KSM test from succeeding deterministically; so drain +* them here (here rather than on entry to ksm_do_scan(), +* so we don't IPI too often when pages_to_scan is set low).
Quoted 8 lines
mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2662
Quoted 5 lines
mm/ksm.c +5
-0+*/ +lru_add_drain_all (); +return true; +} +Can we just leave the
lru_add_drain_all()where it is, or what's the reason for that change?
Sure, There was no special reason for moving lru_add_drain_all()—I just wanted to keep low-level details out of the main scan function to make it look cleaner. Since it is going to be removed soon anyway, leaving it in place makes total sense.
Quoted 8 lines
Note that that might soon get removed either way, so I wouldn't worry about it.
Then, I would rename the function into something like
"
scan_should_start_new_cycle"And keep it simple.
I will rename the helper to scan_should_start_new_cycle(), keep it simple, and send out v2 shortly.