Skip to contentSkip to conversation

Add speculative numa fault support

RFC PATCH New Baolin Wang 1 participant 5 messages Dec 12, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add speculative numa fault support

Hi,

This RFC patch set adds speculative numa fault support for some scenarios, like tiered memory system. On tiered memory system, it will rely on the numa balancing to promote slow and hot memory to fast memory to improve performance. So we can promote several sequential pages on slow memory in advance according to the data locality for some workloads to improve the performance.

So now how much pages need to be promoted to fast memory is the best? Now this RFC patch set only implements a basic and simple mechanism to speculate the numa fault window for each VMA. It will introduce a new atomic member for each VMA to record the numa fault window information, which is used to determine if it is a sequential stream to expand or reduance the numa fault window.

Now I can see about 6% improvement when testing mysql on tiered memory system, more data can be found in patch 1. Looking forword to comments and suggestion to make the algorithm more robust and suitable for more scenarios. Thanks in advance.

Note: this patch set is based on the patch set implemented the tiered memory promotion[1].

[1]
https://lore.kernel.org/lkml/[email protected]/T/

Baolin Wang (4):
mm: Add speculative numa fault support
mm: Add a debug interface to control the range of speculative numa
fault
mm: Add speculative numa fault stats
mm: Update the speculative pages' accessing time

 include/linux/mm_types.h      |   3 +
 include/linux/vm_event_item.h |   1 +
 mm/memory.c                   | 222 ++++++++++++++++++++++++++++++++--
 mm/vmstat.c                   |   1 +
 4 files changed, 216 insertions(+), 11 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

RFC PATCH 1/4 mm: Add speculative numa fault support

Some workloads access a set of data entities will follow the data locality, also known as locality of reference, which means the probability of accessing some data soon after some nearby data has been accessed.

On some systems with different memory types, which will rely on the numa balancing to promote slow hot memory to fast memory to improve performance. So we can promote several sequential pages on slow memory in advance according to the data locality for some workloads to improve the performance.

Thus this patch supports speculative numa fault mechanism to help to migrate suitable pages in advance to improve the performance. And now the basic concept of the speculative numa fault is that, it will add a new member for each VMA to record the numa fault window, which will record the last numa fault address and the pages need to be migrated to the target node. So when numa fault occurs, we will check the last numa fault window for current VMA to check if it is a sequential stream accessing, if yes, we can expand the numa fault window; if not, we can reduce the numa fault winow or close the speculative numa fault to avoid doing unnecessary migration.

Testing with mysql can show about 6% performance improved as below.

Machine: 16 CPUs, 64G DRAM, 256G AEP

sysbench /usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --mysql-user=root --mysql-password=root --oltp-test-mode=complex --oltp-tables-count=80 --oltp-table-size=5000000 --threads=20 --time=600 --report-interval=10 prepare/run

No speculative numa fault:

    queries performed:
        read:                            33039860
        write:                           9439960
        other:                           4719980
        total:                           47199800
    transactions:                        2359990 (3933.28 per sec.)
    queries:                             47199800 (78665.50 per sec.)

Speculative numa fault:

    queries performed:
        read:                            34896862
        write:                           9970532
        other:                           4985266
        total:                           49852660
    transactions:                        2492633 (4154.35 per sec.)
    queries:                             49852660 (83086.94 per sec.)
---
 include/linux/mm_types.h |   3 +
 mm/memory.c              | 165 ++++++++++++++++++++++++++++++++++++---
 2 files changed, 159 insertions(+), 9 deletions(-)
include/linux/mm_types.h +3 -0
index 449b6eafc695..8d8381e9aec9 100644
@@ -474,6 +474,9 @@ struct vm_area_struct {
 #endif
 #ifdef CONFIG_NUMA
 	struct mempolicy *vm_policy;	/* NUMA policy for the VMA */
+#endif
+#ifdef CONFIG_NUMA_BALANCING
+	atomic_long_t numafault_ahead_info;
 #endif
 	struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
 } __randomize_layout;
mm/memory.c +156 -9

218 lines not on this page · raw message

Signature · 1 line
2.27.0
#2 lore raw

RFC PATCH 2/4 mm: Add a debug interface to control the range of speculative numa fault

In reply to Baolin Wang

Add a debug interface to control the range of speculative numa fault, which can be used to tuning the performance or event close the speculative numa fault window for some workloads.

---
 mm/memory.c | 46 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)
mm/memory.c +43 -3
index 2c9ed63e4e23..a0f4a2a008cc 100644
@@ -4052,7 +4052,29 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 static unsigned long fault_around_bytes __read_mostly =
 	rounddown_pow_of_two(65536);
 
+static unsigned long numa_around_bytes __read_mostly;
+
 #ifdef CONFIG_DEBUG_FS
+static int numa_around_bytes_get(void *data, u64 *val)
+{
+	*val = numa_around_bytes;
+	return 0;
+}
+
+static int numa_around_bytes_set(void *data, u64 val)
+{
+	if (val / PAGE_SIZE > PTRS_PER_PTE)
+		return -EINVAL;
+	if (val > PAGE_SIZE)
+		numa_around_bytes = rounddown_pow_of_two(val);
+	else
+		numa_around_bytes = 0; /* rounddown_pow_of_two(0) is undefined */
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(numa_around_bytes_fops,
+			 numa_around_bytes_get,
+			 numa_around_bytes_set, "%llu\n");
+
 static int fault_around_bytes_get(void *data, u64 *val)
 {
 	*val = fault_around_bytes;
@@ -4080,6 +4102,8 @@ static int __init fault_around_debugfs(void)
 {
 	debugfs_create_file_unsafe("fault_around_bytes", 0644, NULL, NULL,
 				   &fault_around_bytes_fops);
+	debugfs_create_file_unsafe("numa_around_bytes", 0644, NULL, NULL,
+				   &numa_around_bytes_fops);
 	return 0;
 }
 late_initcall(fault_around_debugfs);
@@ -4348,10 +4372,13 @@ static bool try_next_numa_page(struct vm_fault *vmf, unsigned int win_pages,
 	((win) & NUMA_FAULT_WINDOW_SIZE_MASK))
 
 static inline unsigned int numa_fault_max_pages(struct vm_area_struct *vma,
-						unsigned long fault_address)
+						unsigned long fault_address,
+						unsigned long numa_around_size)
 {
+	unsigned long numa_around_addr =
+		(fault_address + numa_around_size) & PAGE_MASK;
 	unsigned long pmd_end_addr = (fault_address & PMD_MASK) + PMD_SIZE;
-	unsigned long max_fault_addr = min_t(unsigned long, pmd_end_addr,
+	unsigned long max_fault_addr = min3(numa_around_addr, pmd_end_addr,
 					    vma->vm_end);
 
 	return (max_fault_addr - fault_address - 1) >> PAGE_SHIFT;
@@ -4360,12 +4387,24 @@ static inline unsigned int numa_fault_max_pages(struct vm_area_struct *vma,
 static unsigned int adjust_numa_fault_window(struct vm_area_struct *vma,
 					     unsigned long fault_address)
 {
+	unsigned long numa_around_size = READ_ONCE(numa_around_bytes);
 	unsigned long numafault_ahead = GET_NUMA_FAULT_INFO(vma);
         unsigned long prev_start = NUMA_FAULT_WINDOW_START(numafault_ahead);
         unsigned int prev_pages = NUMA_FAULT_WINDOW_SIZE(numafault_ahead);
 	unsigned long win_start;
 	unsigned int win_pages, max_fault_pages;
 
+	/*
+	 * Shut down the proactive numa fault if the numa_around_bytes
+	 * is set to 0.
+	 */
+	if (!numa_around_size) {
+		if (numafault_ahead)
+			atomic_long_set(&vma->numafault_ahead_info,
+					NUMA_FAULT_INFO(0, 0));
+		return 0;
+	}
+
 	win_start = fault_address + PAGE_SIZE;
 
 	/*
@@ -4437,7 +4476,8 @@ static unsigned int adjust_numa_fault_window(struct vm_area_struct *vma,
 	 * Make sure the size of ahead numa fault address is less than the
 	 * size of current VMA or PMD.
 	 */
-	max_fault_pages = numa_fault_max_pages(vma, fault_address);
+	max_fault_pages = numa_fault_max_pages(vma, fault_address,
+					       numa_around_size);
 	if (win_pages > max_fault_pages)
 		win_pages = max_fault_pages;
 
Signature · 1 line
2.27.0
#3 lore raw

RFC PATCH 3/4 mm: Add speculative numa fault stats

In reply to Baolin Wang

Add a new statistic to help to tune the speculative numa fault window.

---
 include/linux/vm_event_item.h | 1 +
 mm/memory.c                   | 2 ++
 mm/vmstat.c                   | 1 +
 3 files changed, 4 insertions(+)
include/linux/vm_event_item.h +1 -0
index a185cc75ff52..97cdc661b7da 100644
@@ -62,6 +62,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		THP_MIGRATION_SUCCESS,
 		THP_MIGRATION_FAIL,
 		THP_MIGRATION_SPLIT,
+		PGMIGRATE_SPECULATION,
 #endif
 #ifdef CONFIG_COMPACTION
 		COMPACTMIGRATE_SCANNED, COMPACTFREE_SCANNED,
mm/memory.c +2 -0
index a0f4a2a008cc..91122beb6e53 100644
@@ -4572,6 +4572,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	if (migrate_misplaced_page(page, vma, target_nid)) {
 		page_nid = target_nid;
 		flags |= TNF_MIGRATED;
+		if (vmf->address != fault_address)
+			count_vm_events(PGMIGRATE_SPECULATION, 1);
 	} else {
 		flags |= TNF_MIGRATE_FAIL;
 		vmf->pte = pte_offset_map(vmf->pmd, fault_address);
mm/vmstat.c +1 -0
index 787a012de3e2..c64700994786 100644
@@ -1314,6 +1314,7 @@ const char * const vmstat_text[] = {
 	"thp_migration_success",
 	"thp_migration_fail",
 	"thp_migration_split",
+	"pgmigrate_speculation",
 #endif
 #ifdef CONFIG_COMPACTION
 	"compact_migrate_scanned",
Signature · 1 line
2.27.0
#4 lore raw

RFC PATCH 4/4 mm: Update the speculative pages' accessing time

In reply to Baolin Wang

On some systems with different memory types, including fast memory (DRAM) and slow memory (persistent memory), which will rely on the numa balancing to promote slow and hot memory to fast memory to improve performance. After supporting the speculative numa fault, we can update the next pages' accessing time to help to promote it to fast memory node easily to improve the performance.

---
 mm/memory.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
mm/memory.c +13 -2
index 91122beb6e53..e19b10299913 100644
@@ -4556,10 +4556,21 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	 * to record page access time.  So use default value.
 	 */
 	if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
-	    !node_is_toptier(page_nid))
+	    !node_is_toptier(page_nid)) {
 		last_cpupid = (-1 & LAST_CPUPID_MASK);
-	else
+		/*
+		 * According to the data locality for some workloads, the
+		 * probability of accessing some data soon after some nearby
+		 * data has been accessed. So for tiered memory systems, we
+		 * can update the sequential page's age located on slow memory
+		 * type, to try to promote it to fast memory in advance to
+		 * improve the performance.
+		 */
+		if (vmf->address != fault_address)
+			xchg_page_access_time(page, jiffies_to_msecs(jiffies));
+	} else {
 		last_cpupid = page_cpupid_last(page);
+	}
 	target_nid = numa_migrate_prep(page, vma, fault_address, page_nid,
 			&flags);
 	if (target_nid == NUMA_NO_NODE) {
Signature · 1 line
2.27.0