Skip to contentSkip to conversation

memory tiering: read last_cpupid correctly in do_huge_pmd_numa_page()

PATCH Superseded Zi Yan 2 participants 6 messages Jul 19, 2024 – Jul 20, 2024

Subsystems (7)
Memory management MM core Scheduler Memory mapping +3 more subsystems

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Zi Yan

author reviewer
#0 lore raw

PATCH 1/2 memory tiering: read last_cpupid correctly in do_huge_pmd_numa_page()

last_cpupid is only available when memory tiering is off or the folio is in toptier node. Complete the check to read last_cpupid when it is available.

Before the fix, the default last_cpupid will be used even if memory tiering mode is turned off at runtime instead of the actual value. This can prevent task_numa_fault() from getting right numa fault stats, but should not cause any crash. User might see performance changes after the fix.

Reported-by David Hildenbrand <[email protected]>
Closes https://lore.kernel.org/linux-mm/[email protected]/ open here
Fixes 33024536bafd ("memory tiering: hot page selection with hint page fault latency")
Signed-off-by Zi Yan <[email protected]>
Reviewed-by "Huang, Ying" <[email protected]>
Reviewed-by Baolin Wang <[email protected]>
Acked-by David Hildenbrand <[email protected]>
---
 mm/huge_memory.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/huge_memory.c +2 -1
index f4be468e06a4..825317aee88e 100644
@@ -1712,7 +1712,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	 * For memory tiering mode, cpupid of slow memory page is used
 	 * to record page access time.  So use default value.
 	 */
-	if (node_is_toptier(nid))
+	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) ||
+	    node_is_toptier(nid))
 		last_cpupid = folio_last_cpupid(folio);
 	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
 	if (target_nid == NUMA_NO_NODE)
Signature · 1 line
2.43.0
author reviewer
#1 lore raw

PATCH 2/2 memory tiering: introduce folio_has_cpupid() check

Instead of open coded check for if memory tiering mode is on and a folio is in the top tier memory, use a function to encapsulate the check.

Signed-off-by Zi Yan <[email protected]>
Reviewed-by "Huang, Ying" <[email protected]>
---
 include/linux/memory-tiers.h |  8 ++++++++
 kernel/sched/fair.c          |  3 +--
 mm/huge_memory.c             |  6 ++----
 mm/memory-tiers.c            | 17 +++++++++++++++++
 mm/memory.c                  |  3 +--
 mm/mprotect.c                |  3 +--
 6 files changed, 30 insertions(+), 10 deletions(-)
include/linux/memory-tiers.h +8 -0
index 0dc0cf2863e2..10c127d461c4 100644
@@ -73,6 +73,10 @@ static inline bool node_is_toptier(int node)
 }
 #endif
 
+
+bool folio_has_cpupid(struct folio *folio);
+
+
 #else
 
 #define numa_demotion_enabled	false

@@ -151,5 +155,9 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist,
 static inline void mt_put_memory_types(struct list_head *memory_types)
 {
 }
+static inline bool folio_has_cpupid(struct folio *folio)
+{
+	return true;
+}
 #endif	/* CONFIG_NUMA */
 #endif  /* _LINUX_MEMORY_TIERS_H */
kernel/sched/fair.c +1 -2
index 8a5b1ae0aa55..03de808cb3cc 100644
@@ -1840,8 +1840,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
 	 */
-	if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
-	    !node_is_toptier(src_nid)) {
+	if (!folio_has_cpupid(folio)) {
 		struct pglist_data *pgdat;
 		unsigned long rate_limit;
 		unsigned int latency, th, def_th;
mm/huge_memory.c +2 -4
index 825317aee88e..d925a93bb9ed 100644
@@ -1712,8 +1712,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	 * For memory tiering mode, cpupid of slow memory page is used
 	 * to record page access time.  So use default value.
 	 */
-	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) ||
-	    node_is_toptier(nid))
+	if (folio_has_cpupid(folio))
 		last_cpupid = folio_last_cpupid(folio);
 	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
 	if (target_nid == NUMA_NO_NODE)

@@ -2066,8 +2065,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    toptier)
 			goto unlock;
 
-		if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
-		    !toptier)
+		if (!folio_has_cpupid(folio))
 			folio_xchg_access_time(folio,
 					       jiffies_to_msecs(jiffies));
 	}
mm/memory-tiers.c +17 -0
index 4775b3a3dabe..7f0360d4e3a0 100644
@@ -6,6 +6,7 @@
 #include <linux/memory.h>
 #include <linux/memory-tiers.h>
 #include <linux/notifier.h>
+#include <linux/sched/sysctl.h>
 
 #include "internal.h"
 

@@ -50,6 +51,22 @@ static const struct bus_type memory_tier_subsys = {
 	.dev_name = "memory_tier",
 };
 
+/**
+ * folio_has_cpupid - check if a folio has cpupid information
+ * @folio: folio to check
+ *
+ * folio's _last_cpupid field is repurposed by memory tiering. In memory
+ * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
+ * record page access time.
+ *
+ * Return: the folio _last_cpupid is used as cpupid
+ */
+bool folio_has_cpupid(struct folio *folio)
+{
+	return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) ||
+	       node_is_toptier(folio_nid(folio));
+}
+
 #ifdef CONFIG_MIGRATION
 static int top_tier_adistance;
 /*
mm/memory.c +1 -2
index 802d0d8a40f9..105e1a0157dd 100644
@@ -5337,8 +5337,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	 * For memory tiering mode, cpupid of slow memory page is used
 	 * to record page access time.  So use default value.
 	 */
-	if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
-	    !node_is_toptier(nid))
+	if (!folio_has_cpupid(folio))
 		last_cpupid = (-1 & LAST_CPUPID_MASK);
 	else
 		last_cpupid = folio_last_cpupid(folio);
mm/mprotect.c +1 -2
index 222ab434da54..787c3c2bf1b6 100644
@@ -161,8 +161,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 				if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
 				    toptier)
 					continue;
-				if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
-				    !toptier)
+				if (!folio_has_cpupid(folio))
 					folio_xchg_access_time(folio,
 						jiffies_to_msecs(jiffies));
 			}
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 2/2] memory tiering: introduce folio_has_cpupid() check

Quoted 30 lines · Zi Yan

Instead of open coded check for if memory tiering mode is on and a folio is in the top tier memory, use a function to encapsulate the check.

Signed-off-by Zi Yan <[email protected]>
Reviewed-by "Huang, Ying" <[email protected]>
---
  include/linux/memory-tiers.h |  8 ++++++++
  kernel/sched/fair.c          |  3 +--
  mm/huge_memory.c             |  6 ++----
  mm/memory-tiers.c            | 17 +++++++++++++++++
  mm/memory.c                  |  3 +--
  mm/mprotect.c                |  3 +--
  6 files changed, 30 insertions(+), 10 deletions(-)
include/linux/memory-tiers.h +4 -0
index 0dc0cf2863e2..10c127d461c4 100644
@@ -73,6 +73,10 @@ static inline bool node_is_toptier(int node)
  }
  #endif
 
+
+bool folio_has_cpupid(struct folio *folio);
+
+
  #else
 
  #define numa_demotion_enabled	false

include/linux/memory-tiers.h static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist, · line 161

Quoted 7 lines
include/linux/memory-tiers.h +4 -0
@@ -151,5 +155,9 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist,
  static inline void mt_put_memory_types(struct list_head *memory_types)
  {
  }
+static inline bool folio_has_cpupid(struct folio *folio)
+{
+	return true;
+}

Maybe better to move into mm.h since most folio_foo_cpupid()s are there?

author reviewer
#3 lore raw

Re: [PATCH 2/2] memory tiering: introduce folio_has_cpupid() check

Quoted 30 lines · Kefeng Wang
Quoted 29 lines · Zi Yan

Instead of open coded check for if memory tiering mode is on and a folio is in the top tier memory, use a function to encapsulate the check.

Signed-off-by Zi Yan <[email protected]>
Reviewed-by "Huang, Ying" <[email protected]>
---
  include/linux/memory-tiers.h |  8 ++++++++
  kernel/sched/fair.c          |  3 +--
  mm/huge_memory.c             |  6 ++----
  mm/memory-tiers.c            | 17 +++++++++++++++++
  mm/memory.c                  |  3 +--
  mm/mprotect.c                |  3 +--
  6 files changed, 30 insertions(+), 10 deletions(-)
include/linux/memory-tiers.h +3 -0
index 0dc0cf2863e2..10c127d461c4 100644
@@ -73,6 +73,10 @@ static inline bool node_is_toptier(int node)
  }
  #endif
 +
+bool folio_has_cpupid(struct folio *folio);
+
+
  #else
   #define numa_demotion_enabled	false

@@ -151,5 +155,9 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist,
  static inline void mt_put_memory_types(struct list_head *memory_types)
Quoted 8 lines

include/linux/memory-tiers.h static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist, · line 161

Quoted 6 lines
include/linux/memory-tiers.h +4 -0
  {
  }
+static inline bool folio_has_cpupid(struct folio *folio)
+{
+	return true;
+}

Maybe better to move into mm.h since most folio_foo_cpupid()s are there?

Sounds good to me. Will do that in the next version.

--
Best Regards,
Yan, Zi

Reviewed-by on v1 1/2
#4 lore raw
In reply to Zi Yan
Quoted 10 lines · Zi Yan

last_cpupid is only available when memory tiering is off or the folio is in toptier node. Complete the check to read last_cpupid when it is available.

Before the fix, the default last_cpupid will be used even if memory tiering mode is turned off at runtime instead of the actual value. This can prevent task_numa_fault() from getting right numa fault stats, but should not cause any crash. User might see performance changes after the fix.

Quoted 7 lines
Reported-by David Hildenbrand <[email protected]>
Closes https://lore.kernel.org/linux-mm/[email protected]/ open here
Fixes 33024536bafd ("memory tiering: hot page selection with hint page fault latency")
Signed-off-by Zi Yan <[email protected]>
Reviewed-by "Huang, Ying" <[email protected]>
Reviewed-by Baolin Wang <[email protected]>
Acked-by David Hildenbrand <[email protected]>
Reviewed-by Kefeng Wang <[email protected]>

and we better to check numabalance mode in migrate_misplaced_folio()?

mm/migrate.c +2 -1
@@ -2630,7 +2630,8 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
                putback_movable_pages(&migratepages);
        if (nr_succeeded) {
                count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
-               if (!node_is_toptier(folio_nid(folio)) && node_is_toptier(node))
+               if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
+                   &&!node_is_toptier(folio_nid(folio)) && node_is_toptier(node))
                        mod_node_page_state(pgdat, PGPROMOTE_SUCCESS,
                                            nr_succeeded);
        }
Quoted 18 lines
---
  mm/huge_memory.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
mm/huge_memory.c +2 -1
index f4be468e06a4..825317aee88e 100644
@@ -1712,7 +1712,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	 * For memory tiering mode, cpupid of slow memory page is used
  	 * to record page access time.  So use default value.
  	 */
-	if (node_is_toptier(nid))
+	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) ||
+	    node_is_toptier(nid))
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
  	if (target_nid == NUMA_NO_NODE)
author reviewer
#5 lore raw
Quoted 26 lines · Kefeng Wang
Quoted 10 lines · Zi Yan

last_cpupid is only available when memory tiering is off or the folio is in toptier node. Complete the check to read last_cpupid when it is available.

Before the fix, the default last_cpupid will be used even if memory tiering mode is turned off at runtime instead of the actual value. This can prevent task_numa_fault() from getting right numa fault stats, but should not cause any crash. User might see performance changes after the fix.

Quoted 7 lines
Reported-by David Hildenbrand <[email protected]>
Closes https://lore.kernel.org/linux-mm/[email protected]/ open here
Fixes 33024536bafd ("memory tiering: hot page selection with hint page fault latency")
Signed-off-by Zi Yan <[email protected]>
Reviewed-by "Huang, Ying" <[email protected]>
Reviewed-by Baolin Wang <[email protected]>
Acked-by David Hildenbrand <[email protected]>
Reviewed-by Kefeng Wang <[email protected]>

and we better to check numabalance mode in migrate_misplaced_folio()?

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma, · line 2637

Quoted 9 lines
mm/migrate.c +2 -1
@@ -2630,7 +2630,8 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
                putback_movable_pages(&migratepages);
        if (nr_succeeded) {
                count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
-               if (!node_is_toptier(folio_nid(folio)) && node_is_toptier(node))
+               if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
+                   &&!node_is_toptier(folio_nid(folio)) && node_is_toptier(node))
                        mod_node_page_state(pgdat, PGPROMOTE_SUCCESS,
                                            nr_succeeded);
        }

Yes, will add this as a separate fix. Thanks.

--
Best Regards,
Yan, Zi