Skip to contentSkip to conversation

mm/ksm: clean up scan_get_next_rmap_item() initialization

PATCH Superseded Xu Xin (vivo) 4 participants 10 messages Nov 12 – Nov 14

Subsystems (2)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Xu Xin (vivo)

author reviewer
#0 lore raw

cover mm/ksm: clean up scan_get_next_rmap_item() initialization

Currently, scan_get_next_rmap_item() has grown quite large and complex. When introducing new features, such as the only-merge-zero-page mode we are currently working on, we don't want to add more complexity to it and don't want to touch rmap_item structures to avoid unnecessary overhead, so we need to extract some common logic.

With these changes, the entry of scan_get_next_rmap_item() becomes much cleaner and easier to follow, instead of being cluttered with unrelated boilerplate such as stale stable_node pruning and cycle initialization.

Let's start by cleaning up these two parts at the beginning (where I think abstraction is necessary), and we can continue cleaning up further if needed later.

Xu Xin (vivo) (2):
mm/ksm: extract helper functions for scan cycle cleanup mm/ksm: add initialization helpers for scan_get_next_rmap_item()

 mm/ksm.c | 126 ++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 79 insertions(+), 47 deletions(-)
Signature · 1 line
2.34.1
author reviewer
#1 lore raw

PATCH 1/2 mm/ksm: extract helper functions for scan cycle cleanup

Since scan_get_next_rmap_item() is getting too large, extract two new helper functions, ksm_prune_migrate_nodes() and ksm_reset_unstable_trees(), to reduce indentation levels, isolate local variables, and improve the readability of the main scan loop.

No functional change intended.

---
 mm/ksm.c | 56 +++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 33 insertions(+), 23 deletions(-)
mm/ksm.c +33 -23
index 5e24b5ca8dc8..49977bcf04f7 100644
@@ -2609,6 +2609,37 @@ static struct mm_walk_ops ksm_next_page_ops = {
 	.walk_lock = PGWALK_RDLOCK,
 };
 
+static void ksm_prune_migrate_nodes(void)
+{
+	struct ksm_stable_node *stable_node, *next;
+	struct folio *folio;
+
+	if (ksm_merge_across_nodes)
+		return;
+
+	/*
+	 * Whereas stale stable_nodes on the stable_tree itself
+	 * get pruned in the regular course of stable_tree_search(),
+	 * those moved out to the migrate_nodes list can accumulate:
+	 * so prune them once before each full scan.
+	 */
+	list_for_each_entry_safe(stable_node, next, &migrate_nodes, list) {
+		folio = ksm_get_folio(stable_node, KSM_GET_FOLIO_NOLOCK);
+
+		if (folio)
+			folio_put(folio);
+		cond_resched();
+	}
+}
+
+static void ksm_reset_unstable_trees(void)
+{
+	int nid;
+
+	for (nid = 0; nid < ksm_nr_node_ids; nid++)
+		root_unstable_tree[nid] = RB_ROOT;
+}
+
 static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 {
 	struct mm_struct *mm;
@@ -2617,7 +2648,6 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 	struct vm_area_struct *vma;
 	struct ksm_rmap_item *rmap_item;
 	struct vma_iterator vmi;
-	int nid;
 
 	if (list_empty(&ksm_mm_head.slot.mm_node))
 		return NULL;
@@ -2639,28 +2669,8 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 		 */
 		lru_add_drain_all();
 
-		/*
-		 * Whereas stale stable_nodes on the stable_tree itself
-		 * get pruned in the regular course of stable_tree_search(),
-		 * those moved out to the migrate_nodes list can accumulate:
-		 * so prune them once before each full scan.
-		 */
-		if (!ksm_merge_across_nodes) {
-			struct ksm_stable_node *stable_node, *next;
-			struct folio *folio;
-
-			list_for_each_entry_safe(stable_node, next,
-						 &migrate_nodes, list) {
-				folio = ksm_get_folio(stable_node,
-						      KSM_GET_FOLIO_NOLOCK);
-				if (folio)
-					folio_put(folio);
-				cond_resched();
-			}
-		}
-
-		for (nid = 0; nid < ksm_nr_node_ids; nid++)
-			root_unstable_tree[nid] = RB_ROOT;
+		ksm_prune_migrate_nodes();
+		ksm_reset_unstable_trees();
 
 		spin_lock(&ksm_mmlist_lock);
 		slot = list_entry(mm_slot->slot.mm_node.next,
Signature · 1 line
2.34.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] mm/ksm: extract helper functions for scan cycle cleanup

Quoted 8 lines · Xu Xin (vivo)

Since scan_get_next_rmap_item() is getting too large, extract two new helper functions, ksm_prune_migrate_nodes() and ksm_reset_unstable_trees(), to reduce indentation levels, isolate local variables, and improve the readability of the main scan loop.

No functional change intended.

Thanks for doing this! LGTM so:

One small nit below, but it's not a big deal :)

Quoted 8 lines
---
 mm/ksm.c | 56 +++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 33 insertions(+), 23 deletions(-)
mm/ksm.c +0 -0
index 5e24b5ca8dc8..49977bcf04f7 100644

[ ... ]

mm/ksm.c static struct mm_walk_ops ksm_next_page_ops = { · line 2641

Quoted 7 lines
mm/ksm.c +7 -0
+static void ksm_reset_unstable_trees(void)
+{
+	int nid;
+
+	for (nid = 0; nid < ksm_nr_node_ids; nid++)
+		root_unstable_tree[nid] = RB_ROOT;
+}

Recently I learned (and PeterZ reminded me :) that it's actually valid to declare the var in the for-block in linux kernel C.

So this can be:

static void ksm_reset_unstable_trees(void)

{
	for (int nid = 0; nid < ksm_nr_node_ids; nid++)
		root_unstable_tree[nid] = RB_ROOT;
}

:)

--
Cheers, Lorenzo

maintainer Acked-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] mm/ksm: extract helper functions for scan cycle cleanup

In reply to Xu Xin (vivo)
Quoted 9 lines · Xu Xin (vivo)

Since scan_get_next_rmap_item() is getting too large, extract two new helper functions, ksm_prune_migrate_nodes() and ksm_reset_unstable_trees(), to reduce indentation levels, isolate local variables, and improve the readability of the main scan loop.

No functional change intended.

---

Signature · 3 lines
Cheers,

David
author reviewer
#4 lore raw

PATCH 2/2 mm/ksm: add initialization helpers for scan_get_next_rmap_item()

In reply to Xu Xin (vivo)

Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first mm_slot into scan_begin_new_cycle() and scan_get_first_mm().

This keeps the exact execution order intact while making scan_get_next_rmap_item() cleaner and easier to follow.

No functional change intended.

---
 mm/ksm.c | 70 +++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 46 insertions(+), 24 deletions(-)
mm/ksm.c +46 -24
index 49977bcf04f7..e79e64372e49 100644
@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void)
 		root_unstable_tree[nid] = RB_ROOT;
 }
 
+static bool scan_begin_new_cycle(void)
+{
+	/* Check if the cursor points to the head. */
+	if (ksm_scan.mm_slot != &ksm_mm_head)
+		return false;
+
+	/*
+	 * A number of pages can hang around indefinitely in per-cpu
+	 * LRU cache, raised page count preventing write_protect_page
+	 * from merging them.  Though it doesn't really matter much,
+	 * it is puzzling to see some stuck in pages_volatile until
+	 * other activity jostles them out, and they also prevented
+	 * LTP's KSM test from succeeding deterministically; so drain
+	 * them here (here rather than on entry to ksm_do_scan(),
+	 * so we don't IPI too often when pages_to_scan is set low).
+	 */
+	lru_add_drain_all();
+	return true;
+}
+
+static struct ksm_mm_slot *scan_get_first_mm(void)
+{
+	struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot;
+	struct mm_slot *slot;
+
+	spin_lock(&ksm_mmlist_lock);
+
+	slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
+	mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
+	ksm_scan.mm_slot = mm_slot;
+
+	spin_unlock(&ksm_mmlist_lock);
+
+	/*
+	 * Although we tested list_empty() above, a racing __ksm_exit
+	 * of the last mm on the list may have removed it since then.
+	 */
+	if (mm_slot == &ksm_mm_head)
+		return NULL;
+
+	return mm_slot;
+}
+
 static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 {
 	struct mm_struct *mm;
@@ -2653,36 +2696,15 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 		return NULL;
 
 	mm_slot = ksm_scan.mm_slot;
-	if (mm_slot == &ksm_mm_head) {
+	if (scan_begin_new_cycle()) {
 		advisor_start_scan();
 		trace_ksm_start_scan(ksm_scan.seqnr, ksm_rmap_items);
 
-		/*
-		 * A number of pages can hang around indefinitely in per-cpu
-		 * LRU cache, raised page count preventing write_protect_page
-		 * from merging them.  Though it doesn't really matter much,
-		 * it is puzzling to see some stuck in pages_volatile until
-		 * other activity jostles them out, and they also prevented
-		 * LTP's KSM test from succeeding deterministically; so drain
-		 * them here (here rather than on entry to ksm_do_scan(),
-		 * so we don't IPI too often when pages_to_scan is set low).
-		 */
-		lru_add_drain_all();
-
 		ksm_prune_migrate_nodes();
 		ksm_reset_unstable_trees();
 
-		spin_lock(&ksm_mmlist_lock);
-		slot = list_entry(mm_slot->slot.mm_node.next,
-				  struct mm_slot, mm_node);
-		mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
-		ksm_scan.mm_slot = mm_slot;
-		spin_unlock(&ksm_mmlist_lock);
-		/*
-		 * Although we tested list_empty() above, a racing __ksm_exit
-		 * of the last mm on the list may have removed it since then.
-		 */
-		if (mm_slot == &ksm_mm_head)
+		mm_slot = scan_get_first_mm();
+		if (!mm_slot)
 			return NULL;
 next_mm:
 		ksm_scan.address = 0;
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/ksm: scan_begin_new_cycle() reorders lru_add_drain_all() at scan start --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/ksm: scan_begin_new_cycle() reorders lru_add_drain_all() at scan start
--

commit 82208b052cdf6d037a169e81cb68de5a2ad7f1fc
Author: Xu Xin (vivo) <[email protected]>

mm/ksm: add initialization helpers for scan_get_next_rmap_item()

This commit extracts the new scan cycle check and the first mm_slot lookup from scan_get_next_rmap_item() into scan_begin_new_cycle() and scan_get_first_mm().

[Severity: Low]
In mm/ksm.c, does moving lru_add_drain_all() into scan_begin_new_cycle() change the execution order at the start of a scan cycle in scan_get_next_rmap_item()?

Previously, when ksm_scan.mm_slot equaled &ksm_mm_head, advisor_start_scan() and trace_ksm_start_scan() ran before lru_add_drain_all(). Calling lru_add_drain_all() inside scan_begin_new_cycle() in the if condition runs it before both calls, contrary to the commit message's statement that the exact execution order is kept intact.

This reverses the relative order of the mm_lru_add_drain_all and ksm_start_scan tracepoints. It also records advisor_ctx.start_scan via ktime_get() in advisor_start_scan() only after lru_add_drain_all() finishes draining per-CPU LRU caches, whereas scan_time_advisor() still includes the ksmd CPU runtime spent inside lru_add_drain_all() via task_sched_runtime(current).

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()

In reply to Xu Xin (vivo)
Quoted 10 lines · Xu Xin (vivo)

Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first mm_slot into scan_begin_new_cycle() and scan_get_first_mm().

This keeps the exact execution order intact while making scan_get_next_rmap_item() cleaner and easier to follow.

No functional change intended.

Again, thanks for doing this! LGTM and:

Quoted 19 lines
---
 mm/ksm.c | 70 +++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 46 insertions(+), 24 deletions(-)
mm/ksm.c +7 -0
index 49977bcf04f7..e79e64372e49 100644
@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void)
 		root_unstable_tree[nid] = RB_ROOT;
 }
 
+static bool scan_begin_new_cycle(void)
+{
+	/* Check if the cursor points to the head. */
+	if (ksm_scan.mm_slot != &ksm_mm_head)
+		return false;
+
+	/*

mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2659

Quoted 10 lines
mm/ksm.c +10 -0
+	 * A number of pages can hang around indefinitely in per-cpu
+	 * LRU cache, raised page count preventing write_protect_page
+	 * from merging them.  Though it doesn't really matter much,
+	 * it is puzzling to see some stuck in pages_volatile until
+	 * other activity jostles them out, and they also prevented
+	 * LTP's KSM test from succeeding deterministically; so drain
+	 * them here (here rather than on entry to ksm_do_scan(),
+	 * so we don't IPI too often when pages_to_scan is set low).
+	 */
+	lru_add_drain_all();

The sashiko finding is silly, this reorders it after advisor_start_scan() which calls... ktime_get(), yeah it's zero delta :)

Quoted 5 lines
mm/ksm.c +5 -0
+	return true;
+}
+
+static struct ksm_mm_slot *scan_get_first_mm(void)
+{

mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2674

Quoted 10 lines
mm/ksm.c +10 -0
+	struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot;
+	struct mm_slot *slot;
+
+	spin_lock(&ksm_mmlist_lock);
+
+	slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
+	mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
+	ksm_scan.mm_slot = mm_slot;
+
+	spin_unlock(&ksm_mmlist_lock);

NIT: optional, but this could be:

	scoped_guard(mutex, &ksm_mmlist_lock) {
		slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
		mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
		ksm_scan.mm_slot = mm_slot;
	}
Quoted 55 lines
mm/ksm.c +14 -25
+
+	/*
+	 * Although we tested list_empty() above, a racing __ksm_exit
+	 * of the last mm on the list may have removed it since then.
+	 */
+	if (mm_slot == &ksm_mm_head)
+		return NULL;
+
+	return mm_slot;
+}
+
 static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 {
 	struct mm_struct *mm;
@@ -2653,36 +2696,15 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 		return NULL;
 
 	mm_slot = ksm_scan.mm_slot;
-	if (mm_slot == &ksm_mm_head) {
+	if (scan_begin_new_cycle()) {
 		advisor_start_scan();
 		trace_ksm_start_scan(ksm_scan.seqnr, ksm_rmap_items);
 
-		/*
-		 * A number of pages can hang around indefinitely in per-cpu
-		 * LRU cache, raised page count preventing write_protect_page
-		 * from merging them.  Though it doesn't really matter much,
-		 * it is puzzling to see some stuck in pages_volatile until
-		 * other activity jostles them out, and they also prevented
-		 * LTP's KSM test from succeeding deterministically; so drain
-		 * them here (here rather than on entry to ksm_do_scan(),
-		 * so we don't IPI too often when pages_to_scan is set low).
-		 */
-		lru_add_drain_all();
-
 		ksm_prune_migrate_nodes();
 		ksm_reset_unstable_trees();
 
-		spin_lock(&ksm_mmlist_lock);
-		slot = list_entry(mm_slot->slot.mm_node.next,
-				  struct mm_slot, mm_node);
-		mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
-		ksm_scan.mm_slot = mm_slot;
-		spin_unlock(&ksm_mmlist_lock);
-		/*
-		 * Although we tested list_empty() above, a racing __ksm_exit
-		 * of the last mm on the list may have removed it since then.
-		 */
-		if (mm_slot == &ksm_mm_head)
+		mm_slot = scan_get_first_mm();
+		if (!mm_slot)
 			return NULL;
 next_mm:
 		ksm_scan.address = 0;
--
Quoted 2 lines

2.34.1

--
Cheers, Lorenzo

author reviewer
#7 lore raw

Re: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()

Quoted 8 lines · Xu Xin (vivo)
Quoted 7 lines

Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first mm_slot into scan_begin_new_cycle() and scan_get_first_mm().

This keeps the exact execution order intact while making scan_get_next_rmap_item() cleaner and easier to follow.

Quoted 7 lines
Quoted 3 lines

No functional change intended.

Again, thanks for doing this! LGTM and:

Thanks.

Quoted 16 lines

mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2674

Quoted 13 lines
mm/ksm.c +12 -0
+static struct ksm_mm_slot *scan_get_first_mm(void)
+{
+	struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot;
+	struct mm_slot *slot;
+
+	spin_lock(&ksm_mmlist_lock);
+
+	slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
+	mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
+	ksm_scan.mm_slot = mm_slot;
+
+	spin_unlock(&ksm_mmlist_lock);

NIT: optional, but this could be:

Quoted 7 lines
	scoped_guard(mutex, &ksm_mmlist_lock) {
		slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
		mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
		ksm_scan.mm_slot = mm_slot;
	}

Good idea, I will switch to scoped_guard.

#8 lore raw

Re: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()

In reply to Xu Xin (vivo)
Quoted 54 lines · Xu Xin (vivo)

Extract the new scan cycle detection (including draining per-CPU LRU caches) and advancing the scan cursor to the first mm_slot into scan_begin_new_cycle() and scan_get_first_mm().

This keeps the exact execution order intact while making scan_get_next_rmap_item() cleaner and easier to follow.

No functional change intended.

---
 mm/ksm.c | 70 +++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 46 insertions(+), 24 deletions(-)
mm/ksm.c +32 -0
index 49977bcf04f7..e79e64372e49 100644
@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void)
 		root_unstable_tree[nid] = RB_ROOT;
 }
 
+static bool scan_begin_new_cycle(void)
+{
+	/* Check if the cursor points to the head. */
+	if (ksm_scan.mm_slot != &ksm_mm_head)
+		return false;
+
+	/*
+	 * A number of pages can hang around indefinitely in per-cpu
+	 * LRU cache, raised page count preventing write_protect_page
+	 * from merging them.  Though it doesn't really matter much,
+	 * it is puzzling to see some stuck in pages_volatile until
+	 * other activity jostles them out, and they also prevented
+	 * LTP's KSM test from succeeding deterministically; so drain
+	 * them here (here rather than on entry to ksm_do_scan(),
+	 * so we don't IPI too often when pages_to_scan is set low).
+	 */
+	lru_add_drain_all();
+	return true;
+}
+
+static struct ksm_mm_slot *scan_get_first_mm(void)
+{
+	struct ksm_mm_slot *mm_slot = ksm_scan.mm_slot;
+	struct mm_slot *slot;
+
+	spin_lock(&ksm_mmlist_lock);
+
+	slot = list_entry(mm_slot->slot.mm_node.next, struct mm_slot, mm_node);
+	mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
+	ksm_scan.mm_slot = mm_slot;
+
+	spin_unlock(&ksm_mmlist_lock);

mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2684

Quoted 10 lines
mm/ksm.c +10 -0
+
+	/*
+	 * Although we tested list_empty() above, a racing __ksm_exit
+	 * of the last mm on the list may have removed it since then.
+	 */
+	if (mm_slot == &ksm_mm_head)
+		return NULL;
+
+	return mm_slot;
+}

I like this function cleanup here, but not sure about the other one.

Can we just leave the lru_add_drain_all() where it is, or what's the reason for that change?

Note that that might soon get removed either way, so I wouldn't worry about it.

Then, I would rename the function into something like

"scan_should_start_new_cycle"

And keep it simple.

Signature · 3 lines
Cheers,

David
author reviewer
#9 lore raw

Re: [PATCH 2/2] mm/ksm: add initialization helpers for scan_get_next_rmap_item()

Quoted 23 lines
Quoted 23 lines
mm/ksm.c +15 -0
index 49977bcf04f7..e79e64372e49 100644
@@ -2640,6 +2640,49 @@ static void ksm_reset_unstable_trees(void)
 		root_unstable_tree[nid] = RB_ROOT;
 }
 
+static bool scan_begin_new_cycle(void)
+{
+	/* Check if the cursor points to the head. */
+	if (ksm_scan.mm_slot != &ksm_mm_head)
+		return false;
+
+	/*
+	 * A number of pages can hang around indefinitely in per-cpu
+	 * LRU cache, raised page count preventing write_protect_page
+	 * from merging them.  Though it doesn't really matter much,
+	 * it is puzzling to see some stuck in pages_volatile until
+	 * other activity jostles them out, and they also prevented
+	 * LTP's KSM test from succeeding deterministically; so drain
+	 * them here (here rather than on entry to ksm_do_scan(),
+	 * so we don't IPI too often when pages_to_scan is set low).
Quoted 8 lines

mm/ksm.c static void ksm_reset_unstable_trees(void) · line 2662

Quoted 5 lines
mm/ksm.c +5 -0
+	 */
+	lru_add_drain_all();
+	return true;
+}
+

Can we just leave the lru_add_drain_all() where it is, or what's the reason for that change?

Sure, There was no special reason for moving lru_add_drain_all()—I just wanted to keep low-level details out of the main scan function to make it look cleaner. Since it is going to be removed soon anyway, leaving it in place makes total sense.

Quoted 8 lines

Note that that might soon get removed either way, so I wouldn't worry about it.

Then, I would rename the function into something like

"scan_should_start_new_cycle"

And keep it simple.

I will rename the helper to scan_should_start_new_cycle(), keep it simple, and send out v2 shortly.