mm/ksm: break_ksm() cleanups and fixes
Subsystems (6)
v2 cover mm/ksm: break_ksm() cleanups and fixes
This series cleans up and fixes break_ksm(). In summary, we no longer use fake write faults to break COW but instead FAULT_FLAG_UNSHARE. Further, we move away from using follow_page() --- that we can hopefully remove completely at one point --- and use new walk_page_range_vma() instead.
Fortunately, we can get rid of VM_FAULT_WRITE and FOLL_MIGRATION in common code now.
Extend the existing ksm tests by an unmerge benchmark, and a some new unmerge tests.
Add a selftest to measure MADV_UNMERGEABLE performance. In my setup (AMD Ryzen 9 3900X), running the KSM selftest to test unmerge performance on 2 GiB (taskset 0x8 ./ksm_tests -D -s 2048), this results in a performance degradation of ~6% -- 7% (old: ~5250 MiB/s, new: ~4900 MiB/s). I don't think we particularly care for now, but it's good to be aware of the implication.
v1 -> v2:
* "selftests/vm: add KSM unmerge tests"
-> Add new unmerge tests
* "mm/ksm: fix KSM COW breaking with userfaultfd-wp via FAULT_FLAG_UNSHARE" -> Simplify patch description now that we have a selftest
* "mm/pagewalk: don't trigger test_walk() in walk_page_vma()" -> Added
* "mm/pagewalk: add walk_page_range_vma()"
-> Don't call test_walk()
* "mm/ksm: convert break_ksm() to use walk_page_range_vma()"
-> Simplify and fix missing unlock, fix missing "static"
David Hildenbrand (9):
selftests/vm: add test to measure MADV_UNMERGEABLE performance mm/ksm: simplify break_ksm() to not rely on VM_FAULT_WRITE
mm: remove VM_FAULT_WRITE
selftests/vm: add KSM unmerge tests
mm/ksm: fix KSM COW breaking with userfaultfd-wp via
FAULT_FLAG_UNSHARE
mm/pagewalk: don't trigger test_walk() in walk_page_vma() mm/pagewalk: add walk_page_range_vma()
mm/ksm: convert break_ksm() to use walk_page_range_vma() mm/gup: remove FOLL_MIGRATION
include/linux/mm.h | 1 - include/linux/mm_types.h | 3 - include/linux/pagewalk.h | 5 + mm/gup.c | 55 +--- mm/huge_memory.c | 2 +- mm/ksm.c | 78 +++-- mm/memory.c | 9 +- mm/pagewalk.c | 27 +- tools/testing/selftests/vm/Makefile | 2 + .../selftests/vm/ksm_functional_tests.c | 279 ++++++++++++++++++ tools/testing/selftests/vm/ksm_tests.c | 76 ++++- tools/testing/selftests/vm/run_vmtests.sh | 2 + tools/testing/selftests/vm/vm_util.c | 10 + tools/testing/selftests/vm/vm_util.h | 1 + 14 files changed, 456 insertions(+), 94 deletions(-) create mode 100644 tools/testing/selftests/vm/ksm_functional_tests.c
base-commit: 9abf2313adc1ca1b6180c508c25f22f9395cc780
Signature · 1 line
2.37.3
PATCH v2 1/9 selftests/vm: add test to measure MADV_UNMERGEABLE performance
Let's add a test to measure performance of KSM breaking not triggered via COW, but triggered by disabling KSM on an area filled with KSM pages via MADV_UNMERGEABLE.
--- tools/testing/selftests/vm/ksm_tests.c | 76 +++++++++++++++++++++++++- 1 file changed, 74 insertions(+), 2 deletions(-)
tools/testing/selftests/vm/ksm_tests.c +74 -2
@@ -40,6 +40,7 @@ enum ksm_test_name {CHECK_KSM_NUMA_MERGE ,KSM_MERGE_TIME ,KSM_MERGE_TIME_HUGE_PAGES , +KSM_UNMERGE_TIME , KSM_COW_TIME };
@@ -108,7 +109,10 @@ static void print_help(void)" -P evaluate merging time and speed.\n" " For this test, the size of duplicated memory area (in MiB)\n" " must be provided using -s option\n" -" -H evaluate merging time and speed of area allocated mostly with huge pages\n" +" -H evaluate merging time and speed of area allocated mostly with huge pages\n" +" For this test, the size of duplicated memory area (in MiB)\n" +" must be provided using -s option\n" +" -D evaluate unmerging time and speed when disabling KSM.\n" " For this test, the size of duplicated memory area (in MiB)\n" " must be provided using -s option\n" " -C evaluate the time required to break COW of merged pages.\n\n");
@@ -188,6 +192,16 @@ static int ksm_merge_pages(void *addr, size_t size, struct timespec start_time,return 0 ; } +static int ksm_unmerge_pages (void *addr,size_t size, +struct timespec start_time,int timeout) +{ +if (madvise (addr, size,MADV_UNMERGEABLE )) { +perror ("madvise" ); +return 1 ; + } +return 0 ; +} +static bool assert_ksm_pages_count (long dupl_page_count) {unsigned long max_page_sharing, pages_sharing, pages_shared;
@@ -560,6 +574,53 @@ static int ksm_merge_time(int mapping, int prot, int timeout, size_t map_size)return KSFT_FAIL ; } +static int ksm_unmerge_time (int mapping,int prot,int timeout,size_t map_size) +{ +void *map_ptr; +struct timespec start_time, end_time; +unsigned long scan_time_ns; + + map_size *=MB ; + + map_ptr =allocate_memory (NULL , prot, mapping,'*' , map_size); +if (!map_ptr) +return KSFT_FAIL ; +if (clock_gettime (CLOCK_MONOTONIC_RAW , &start_time)) { +perror ("clock_gettime" ); + goto err_out; + } +if (ksm_merge_pages (map_ptr, map_size, start_time, timeout)) + goto err_out; + +if (clock_gettime (CLOCK_MONOTONIC_RAW , &start_time)) { +perror ("clock_gettime" ); + goto err_out; + } +if (ksm_unmerge_pages (map_ptr, map_size, start_time, timeout)) + goto err_out; +if (clock_gettime (CLOCK_MONOTONIC_RAW , &end_time)) { +perror ("clock_gettime" ); + goto err_out; + } + + scan_time_ns = (end_time.tv_sec - start_time.tv_sec) *NSEC_PER_SEC + + (end_time.tv_nsec - start_time.tv_nsec); + +printf ("Total size: %lu MiB\n" , map_size /MB ); +printf ("Total time: %ld.%09ld s\n" , scan_time_ns /NSEC_PER_SEC , + scan_time_ns %NSEC_PER_SEC ); +printf ("Average speed: %.3f MiB/s\n" , (map_size /MB ) / + ((double )scan_time_ns /NSEC_PER_SEC )); + +munmap (map_ptr, map_size); +return KSFT_PASS ; + +err_out: +printf ("Not OK\n" ); +munmap (map_ptr, map_size); +return KSFT_FAIL ; +} +static int ksm_cow_time (int mapping,int prot,int timeout,size_t page_size) {void *map_ptr;
@@ -644,7 +705,7 @@ int main(int argc, char *argv[])bool merge_across_nodes =KSM_MERGE_ACROSS_NODES_DEFAULT ;long size_MB =0 ; -while ((opt =getopt (argc, argv,"ha:p:l:z:m:s:MUZNPCH" )) !=-1 ) { +while ((opt =getopt (argc, argv,"ha:p:l:z:m:s:MUZNPCHD" )) !=-1 ) {switch (opt) {case 'a' : prot =str_to_prot (optarg);
@@ -701,6 +762,9 @@ int main(int argc, char *argv[])case 'H' : test_name =KSM_MERGE_TIME_HUGE_PAGES ;break ; +case 'D' : + test_name =KSM_UNMERGE_TIME ; +break ;case 'C' : test_name =KSM_COW_TIME ;break ;
@@ -762,6 +826,14 @@ int main(int argc, char *argv[]) ret =ksm_merge_hugepages_time (MAP_PRIVATE |MAP_ANONYMOUS , prot, ksm_scan_limit_sec, size_MB);break ; +case KSM_UNMERGE_TIME : +if (size_MB ==0 ) { +printf ("Option '-s' is required.\n" ); +return KSFT_FAIL ; + } + ret =ksm_unmerge_time (MAP_PRIVATE |MAP_ANONYMOUS , prot, + ksm_scan_limit_sec, size_MB); +break ;case KSM_COW_TIME : ret =ksm_cow_time (MAP_PRIVATE |MAP_ANONYMOUS , prot, ksm_scan_limit_sec, page_size);
Signature · 1 line
2.37.3
PATCH v2 2/9 mm/ksm: simplify break_ksm() to not rely on VM_FAULT_WRITE
In reply to David HildenbrandNow that GUP no longer requires VM_FAULT_WRITE, break_ksm() is the sole remaining user of VM_FAULT_WRITE. As we also want to stop triggering a fake write fault and instead use FAULT_FLAG_UNSHARE -- similar to GUP-triggered unsharing when taking a R/O pin on a shared anonymous page (including KSM pages), let's stop relying on VM_FAULT_WRITE.
Let's rework break_ksm() to not rely on the return value of handle_mm_fault() anymore to figure out whether COW-breaking was successful. Simply perform another follow_page() lookup to verify the result.
While this makes break_ksm() slightly less efficient, we can simplify handle_mm_fault() a little and easily switch to FAULT_FLAG_UNSHARE without introducing similar KSM-specific behavior for FAULT_FLAG_UNSHARE.
In my setup (AMD Ryzen 9 3900X), running the KSM selftest to test unmerge performance on 2 GiB (taskset 0x8 ./ksm_tests -D -s 2048), this results in a performance degradation of ~4% -- 5% (old: ~5250 MiB/s,
new: ~5010 MiB/s).
I don't think that we particularly care about that performance drop when unmerging. If it ever turns out to be an actual performance issue, we can think about a better alternative for FAULT_FLAG_UNSHARE -- let's just keep it simple for now.
--- mm/ksm.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
mm/ksm.c +13 -12
@@ -440,26 +440,27 @@ static int break_ksm(struct vm_area_struct *vma, unsigned long addr)vm_fault_t ret =0 ;do { +bool ksm_page = false; +cond_resched (); page =follow_page (vma, addr,FOLL_GET |FOLL_MIGRATION |FOLL_REMOTE );if (IS_ERR_OR_NULL (page))break ;if (PageKsm (page)) - ret =handle_mm_fault (vma, addr, -FAULT_FLAG_WRITE |FAULT_FLAG_REMOTE , -NULL ); -else - ret =VM_FAULT_WRITE ; + ksm_page = true;put_page (page); - }while (!(ret & (VM_FAULT_WRITE |VM_FAULT_SIGBUS |VM_FAULT_SIGSEGV |VM_FAULT_OOM ))); + +if (!ksm_page) +return 0 ; + ret =handle_mm_fault (vma, addr, +FAULT_FLAG_WRITE |FAULT_FLAG_REMOTE , +NULL ); + }while (!(ret & (VM_FAULT_SIGBUS |VM_FAULT_SIGSEGV |VM_FAULT_OOM )));/* -* We must loop because handle_mm_fault() may back out if there's -* any difficulty e.g. if pte accessed bit gets updated concurrently. -* -* VM_FAULT_WRITE is what we have been hoping for: it indicates that -* COW has been broken, even if the vma does not permit VM_WRITE; -* but note that a concurrent fault might break PageKsm for us. +* We must loop until we no longer find a KSM page because +* handle_mm_fault() may back out if there's any difficulty e.g. if +* pte accessed bit gets updated concurrently. * * VM_FAULT_SIGBUS could occur if we race with truncation of the * backing file, which also invalidates anonymous pages: that's
Signature · 1 line
2.37.3
PATCH v2 3/9 mm: remove VM_FAULT_WRITE
In reply to David HildenbrandAll users -- GUP and KSM -- are gone, let's just remove it.
--- include/linux/mm_types.h | 3 --- mm/huge_memory.c | 2 +- mm/memory.c | 9 ++++----- 3 files changed, 5 insertions(+), 9 deletions(-)
include/linux/mm_types.h +0 -3
@@ -847,7 +847,6 @@ typedef __bitwise unsigned int vm_fault_t;* @VM_FAULT_OOM: Out Of Memory * @VM_FAULT_SIGBUS: Bad access * @VM_FAULT_MAJOR: Page read from storage -* @VM_FAULT_WRITE: Special case for get_user_pages * @VM_FAULT_HWPOISON: Hit poisoned small page * @VM_FAULT_HWPOISON_LARGE: Hit poisoned large page. Index encoded * in upper bits
@@ -868,7 +867,6 @@ enum vm_fault_reason {VM_FAULT_OOM = (__force vm_fault_t )0x000001 ,VM_FAULT_SIGBUS = (__force vm_fault_t )0x000002 ,VM_FAULT_MAJOR = (__force vm_fault_t )0x000004 , -VM_FAULT_WRITE = (__force vm_fault_t )0x000008 ,VM_FAULT_HWPOISON = (__force vm_fault_t )0x000010 ,VM_FAULT_HWPOISON_LARGE = (__force vm_fault_t )0x000020 , VM_FAULT_SIGSEGV = (__force vm_fault_t)0x000040,
@@ -894,7 +892,6 @@ enum vm_fault_reason { { VM_FAULT_OOM, "OOM" }, \ { VM_FAULT_SIGBUS, "SIGBUS" }, \ { VM_FAULT_MAJOR, "MAJOR" }, \ - { VM_FAULT_WRITE, "WRITE" }, \ { VM_FAULT_HWPOISON, "HWPOISON" }, \ { VM_FAULT_HWPOISON_LARGE, "HWPOISON_LARGE" }, \ { VM_FAULT_SIGSEGV, "SIGSEGV" }, \
mm/huge_memory.c +1 -1
@@ -1379,7 +1379,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);spin_unlock (vmf->ptl); -return VM_FAULT_WRITE ; +return 0 ; } unlock_fallback:
mm/memory.c +4 -5
@@ -3242,7 +3242,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) }delayacct_wpcopy_end (); -return (page_copied && !unshare) ?VM_FAULT_WRITE :0 ; +return 0 ; oom_free_new:put_page (new_page); oom:
@@ -3306,14 +3306,14 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf)return finish_mkwrite_fault (vmf); }wp_page_reuse (vmf); -return VM_FAULT_WRITE ; +return 0 ; }static vm_fault_t wp_page_shared (struct vm_fault *vmf) __releases(vmf->ptl) {struct vm_area_struct *vma = vmf->vma; -vm_fault_t ret =VM_FAULT_WRITE ; +vm_fault_t ret =0 ;get_page (vmf->page);
@@ -3464,7 +3464,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)return 0 ; }wp_page_reuse (vmf); -return VM_FAULT_WRITE ; +return 0 ; }else if (unshare) {/* No anonymous page -> nothing to do. */ pte_unmap_unlock (vmf->pte, vmf->ptl);
@@ -3983,7 +3983,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; - ret |=VM_FAULT_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; }
Signature · 1 line
2.37.3
PATCH v2 4/9 selftests/vm: add KSM unmerge tests
In reply to David HildenbrandLet's add three unmerge tests (MADV_UNMERGEABLE unmerging all pages in the range).
test_unmerge(): basic unmerge tests test_unmerge_discarded(): have some pte_none() entries in the range test_unmerge_uffd_wp(): protect the merged pages using uffd-wp
ksm_tests.c currently contains a mixture of benchmarks and tests, whereby each test is carried out by executing the ksm_tests binary with specific parameters. Let's add new ksm_functional_tests.c that performs multiple, smaller functional tests all at once.
--- tools/testing/selftests/vm/Makefile | 2 + .../selftests/vm/ksm_functional_tests.c | 279 ++++++++++++++++++ tools/testing/selftests/vm/run_vmtests.sh | 2 + tools/testing/selftests/vm/vm_util.c | 10 + tools/testing/selftests/vm/vm_util.h | 1 + 5 files changed, 294 insertions(+) create mode 100644 tools/testing/selftests/vm/ksm_functional_tests.c
tools/testing/selftests/vm/Makefile +2 -0
@@ -52,6 +52,7 @@ TEST_GEN_FILES += userfaultfd TEST_GEN_PROGS += soft-dirty TEST_GEN_PROGS += split_huge_page_test TEST_GEN_FILES += ksm_tests +TEST_GEN_PROGS += ksm_functional_tests ifeq ($(MACHINE),x86_64) CAN_BUILD_I386 := $(shell ./../x86/check_cc.sh "$(CC)" ../x86/trivial_32bit_program.c -m32)
@@ -96,6 +97,7 @@ TEST_FILES += va_128TBswitch.sh include ../lib.mk $(OUTPUT)/khugepaged: vm_util.c +$(OUTPUT)/ksm_functional_tests: vm_util.c $(OUTPUT)/madv_populate: vm_util.c $(OUTPUT)/soft-dirty: vm_util.c $(OUTPUT)/split_huge_page_test: vm_util.c
tools/testing/selftests/vm/ksm_functional_tests.c new file +279 -0
279 lines not on this page · raw message
tools/testing/selftests/vm/run_vmtests.sh +2 -0
@@ -184,6 +184,8 @@ run_test ./ksm_tests -N -m 1 # KSM test with 2 NUMA nodes and merge_across_nodes = 0 run_test ./ksm_tests -N -m 0 +run_test ./ksm_functional_tests + # protection_keys tests if [ -x ./protection_keys_32 ] then
tools/testing/selftests/vm/vm_util.c +10 -0
@@ -28,6 +28,16 @@ bool pagemap_is_softdirty(int fd, char *start)return entry &0x0080000000000000ull ; } +unsigned long pagemap_get_pfn (int fd,char *start) +{ +uint64_t entry =pagemap_get_entry (fd, start); + +/* If present (63th bit), PFN is at bit 0 -- 54. */ +if (entry &0x8000000000000000ull ) +return entry &0x007fffffffffffffull ; +return -1ull ; +} +void clear_softdirty (void ) {int ret;
tools/testing/selftests/vm/vm_util.h +1 -0
@@ -4,6 +4,7 @@uint64_t pagemap_get_entry (int fd,char *start);bool pagemap_is_softdirty (int fd,char *start); +unsigned long pagemap_get_pfn (int fd,char *start);void clear_softdirty (void );bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t read_pmd_pagesize (void );
Signature · 1 line
2.37.3
PATCH v2 5/9 mm/ksm: fix KSM COW breaking with userfaultfd-wp via FAULT_FLAG_UNSHARE
In reply to David HildenbrandLet's stop breaking COW via a fake write fault and let's use FAULT_FLAG_UNSHARE instead. This avoids any wrong side effects of the fake write fault, such as mapping the PTE writable and marking the pte dirty/softdirty.
Consequently, we will no longer trigger a fake write fault and break COW without any such side-effects.
Also, this fixes KSM interaction with userfaultfd-wp: when we have a KSM page that's write-protected by userfaultfd, break_ksm()->handle_mm_fault() will fail with VM_FAULT_SIGBUS and will simply return in break_ksm() with 0 instead of actually breaking COW.
For now, the KSM unmerge tests can trigger that:
$ sudo ./ksm_functional_tests
TAP version 13
1..3
# [RUN] test_unmerge
ok 1 Pages were unmerged
# [RUN] test_unmerge_discarded
ok 2 Pages were unmerged
# [RUN] test_unmerge_uffd_wp
not ok 3 Pages were unmerged
Bail out! 1 out of 3 tests failed
# Planned tests != run tests (2 != 3)
# Totals: pass:2 fail:1 xfail:0 xpass:0 skip:0 error:0The warning in dmesg also indicates this wrong handling:
[ 230.096368] FAULT_FLAG_ALLOW_RETRY missing 881
[ 230.100822] CPU: 1 PID: 1643 Comm: ksm-uffd-wp [...]
[ 230.110124] Hardware name: [...]
[ 230.117775] Call Trace:
[ 230.120227] <TASK>
[ 230.122334] dump_stack_lvl+0x44/0x5c
[ 230.126010] handle_userfault.cold+0x14/0x19
[ 230.130281] ? tlb_finish_mmu+0x65/0x170
[ 230.134207] ? uffd_wp_range+0x65/0xa0
[ 230.137959] ? _raw_spin_unlock+0x15/0x30
[ 230.141972] ? do_wp_page+0x50/0x590
[ 230.145551] __handle_mm_fault+0x9f5/0xf50
[ 230.149652] ? mmput+0x1f/0x40
[ 230.152712] handle_mm_fault+0xb9/0x2a0
[ 230.156550] break_ksm+0x141/0x180
[ 230.159964] unmerge_ksm_pages+0x60/0x90
[ 230.163890] ksm_madvise+0x3c/0xb0
[ 230.167295] do_madvise.part.0+0x10c/0xeb0
[ 230.171396] ? do_syscall_64+0x67/0x80
[ 230.175157] __x64_sys_madvise+0x5a/0x70
[ 230.179082] do_syscall_64+0x58/0x80
[ 230.182661] ? do_syscall_64+0x67/0x80
[ 230.186413] entry_SYSCALL_64_after_hwframe+0x63/0xcdThis is primarily a fix for KSM+userfaultfd-wp, however, the fake write fault was always questionable. As this fix is not easy to backport and it's not very critical, let's not cc stable.
--- mm/ksm.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
mm/ksm.c +5 -7
@@ -420,17 +420,15 @@ static inline bool ksm_test_exit(struct mm_struct *mm) }/* -* We use break_ksm to break COW on a ksm page: it's a stripped down +* We use break_ksm to break COW on a ksm page by triggering unsharing, +* such that the ksm page will get replaced by an exclusive anonymous page. * -* if (get_user_pages(addr, 1, FOLL_WRITE, &page, NULL) == 1) -* put_page(page); -* -* but taking great care only to touch a ksm page, in a VM_MERGEABLE vma, +* We take great care only to touch a ksm page, in a VM_MERGEABLE vma, * in case the application has unmapped and remapped mm,addr meanwhile. * Could a ksm page appear anywhere else? Actually yes, in a VM_PFNMAP * mmap of /dev/mem, where we would not want to touch it. * -* FAULT_FLAG/FOLL_REMOTE are because we do this outside the context +* FAULT_FLAG_REMOTE/FOLL_REMOTE are because we do this outside the context * of the process that owns 'vma'. We also do not want to enforce * protection keys here anyway. */
@@ -454,7 +452,7 @@ static int break_ksm(struct vm_area_struct *vma, unsigned long addr)if (!ksm_page)return 0 ; ret =handle_mm_fault (vma, addr, -FAULT_FLAG_WRITE |FAULT_FLAG_REMOTE , +FAULT_FLAG_UNSHARE |FAULT_FLAG_REMOTE ,NULL ); }while (!(ret & (VM_FAULT_SIGBUS |VM_FAULT_SIGSEGV |VM_FAULT_OOM )));/*
Signature · 1 line
2.37.3
PATCH v2 6/9 mm/pagewalk: don't trigger test_walk() in walk_page_vma()
In reply to David HildenbrandAs Peter points out, the caller passes a single VMA and can just do that check itself.
And in fact, no existing users rely on test_walk() getting called. So let's just remove it and make the implementation slightly more efficient.
--- include/linux/pagewalk.h | 2 ++ mm/pagewalk.c | 7 ------- 2 files changed, 2 insertions(+), 7 deletions(-)
include/linux/pagewalk.h +2 -0
@@ -27,6 +27,8 @@ struct mm_walk;* "do page table walk over the current vma", returning * a negative value means "abort current page table walk * right now" and returning 1 means "skip the current vma" +* Note that this callback is not called when the caller +* passes in a single VMA as for walk_page_vma(). * @pre_vma: if set, called before starting walk on a non-null vma. * @post_vma: if set, called after a walk on a non-null vma, provided * that @pre_vma and the vma walk succeeded.
mm/pagewalk.c +0 -7
@@ -526,18 +526,11 @@ int walk_page_vma(struct vm_area_struct *vma, const struct mm_walk_ops *ops, .vma = vma, .private = private, }; -int err;if (!walk.mm)return -EINVAL ;mmap_assert_locked (walk.mm); - - err =walk_page_test (vma->vm_start, vma->vm_end, &walk); -if (err >0 ) -return 0 ; -if (err <0 ) -return err;return __walk_page_range (vma->vm_start, vma->vm_end, &walk); }
Signature · 1 line
2.37.3
PATCH v2 7/9 mm/pagewalk: add walk_page_range_vma()
In reply to David HildenbrandLet's add walk_page_range_vma(), which is similar to walk_page_vma(), however, is only interested in a subset of the VMA range.
To be used in KSM code to stop using follow_page() next.
--- include/linux/pagewalk.h | 3 +++ mm/pagewalk.c | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+)
include/linux/pagewalk.h +3 -0
@@ -101,6 +101,9 @@ int walk_page_range_novma(struct mm_struct *mm, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, pgd_t *pgd, void *private); +int walk_page_range_vma (struct vm_area_struct *vma,unsigned long start, +unsigned long end,const struct mm_walk_ops *ops, +void *private);int walk_page_vma (struct vm_area_struct *vma,const struct mm_walk_ops *ops,void *private); int walk_page_mapping(struct address_space *mapping, pgoff_t first_index,
mm/pagewalk.c +20 -0
@@ -517,6 +517,26 @@ int walk_page_range_novma(struct mm_struct *mm, unsigned long start,return walk_pgd_range (start, end, &walk); } +int walk_page_range_vma (struct vm_area_struct *vma,unsigned long start, +unsigned long end,const struct mm_walk_ops *ops, +void *private) +{ +struct mm_walk walk = { + .ops = ops, + .mm = vma->vm_mm, + .vma = vma, + .private = private, + }; + +if (start >= end || !walk.mm) +return -EINVAL ; +if (start < vma->vm_start || end > vma->vm_end) +return -EINVAL ; + +mmap_assert_locked (walk.mm); +return __walk_page_range (start, end, &walk); +} +int walk_page_vma (struct vm_area_struct *vma,const struct mm_walk_ops *ops,void *private) {
Signature · 1 line
2.37.3
PATCH v2 8/9 mm/ksm: convert break_ksm() to use walk_page_range_vma()
In reply to David HildenbrandFOLL_MIGRATION exists only for the purpose of break_ksm(), and actually, there is not even the need to wait for the migration to finish, we only want to know if we're dealing with a KSM page.
Using follow_page() just to identify a KSM page overcomplicates GUP code. Let's use walk_page_range_vma() instead, because we don't actually care about the page itself, we only need to know a single property -- no need to even grab a reference.
So, get rid of follow_page() usage such that we can get rid of FOLL_MIGRATION now and eventually be able to get rid of follow_page() in the future.
In my setup (AMD Ryzen 9 3900X), running the KSM selftest to test unmerge performance on 2 GiB (taskset 0x8 ./ksm_tests -D -s 2048), this results in a performance degradation of ~2% (old: ~5010 MiB/s, new: ~4900 MiB/s). I don't think we particularly care for now.
Interestingly, the benchmark reduction is due to the single callback. Adding a second callback (e.g., pud_entry()) reduces the benchmark by another 100-200 MiB/s.
--- mm/ksm.c | 49 +++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 10 deletions(-)
mm/ksm.c +39 -10
@@ -39,6 +39,7 @@#include <linux/freezer.h> #include <linux/oom.h> #include <linux/numa.h> +#include <linux/pagewalk.h> #include <asm/tlbflush.h> #include "internal.h"
@@ -419,6 +420,39 @@ static inline bool ksm_test_exit(struct mm_struct *mm)return atomic_read (&mm->mm_users) ==0 ; } +static int break_ksm_pmd_entry (pmd_t *pmd,unsigned long addr,unsigned long next, +struct mm_walk *walk) +{ +struct page *page =NULL ; +spinlock_t *ptl; +pte_t *pte; +int ret; + +if (pmd_leaf (*pmd) || !pmd_present (*pmd)) +return 0 ; + + pte =pte_offset_map_lock (walk->mm, pmd, addr, &ptl); +if (pte_present (*pte)) { + page =vm_normal_page (walk->vma, addr, *pte); + }else if (!pte_none (*pte)) { +swp_entry_t entry =pte_to_swp_entry (*pte); + +/* +* As KSM pages remain KSM pages until freed, no need to wait +* here for migration to end. +*/ +if (is_migration_entry (entry)) + page =pfn_swap_entry_to_page (entry); + } + ret = page &&PageKsm (page); +pte_unmap_unlock (pte, ptl); +return ret; +} + +static const struct mm_walk_ops break_ksm_ops = { + .pmd_entry = break_ksm_pmd_entry, +}; +/* * We use break_ksm to break COW on a ksm page by triggering unsharing, * such that the ksm page will get replaced by an exclusive anonymous page.
@@ -434,21 +468,16 @@ static inline bool ksm_test_exit(struct mm_struct *mm)*/ static int break_ksm(struct vm_area_struct *vma, unsigned long addr) { - struct page *page; vm_fault_t ret = 0; do { - bool ksm_page = false; + int ksm_page; cond_resched(); - page = follow_page(vma, addr, - FOLL_GET | FOLL_MIGRATION | FOLL_REMOTE); - if (IS_ERR_OR_NULL(page)) - break; - if (PageKsm(page)) - ksm_page = true; - put_page(page); - + ksm_page = walk_page_range_vma(vma, addr, addr + 1, + &break_ksm_ops, NULL); + if (WARN_ON_ONCE(ksm_page < 0)) + return ksm_page; if (!ksm_page) return 0; ret = handle_mm_fault(vma, addr,
Signature · 1 line
2.37.3
PATCH v2 9/9 mm/gup: remove FOLL_MIGRATION
In reply to David HildenbrandFortunately, the last user (KSM) is gone, so let's just remove this rather special code from generic GUP handling -- especially because KSM never required the PMD handling as KSM only deals with individual base pages.
--- include/linux/mm.h | 1 - mm/gup.c | 55 +++++----------------------------------------- 2 files changed, 5 insertions(+), 51 deletions(-)
include/linux/mm.h +0 -1
@@ -2950,7 +2950,6 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,* and return without waiting upon it */ #define FOLL_NOFAULT 0x80/* do not fault in pages */ #define FOLL_HWPOISON 0x100/* check page is hwpoisoned */ -#define FOLL_MIGRATION 0x400/* wait for page to replace migration entry */ #define FOLL_TRIED 0x800/* a retry, previous pass started an IO */ #define FOLL_REMOTE 0x2000/* we are working on non-current tsk/mm */ #define FOLL_ANON 0x8000/* don't do file mappings */
mm/gup.c +5 -50
81 lines not on this page · raw message
Signature · 1 line
2.37.3
Andrew Morton
maintainerQuoted 1 line · On Fri, 21 Oct 2022 12:11:32 +0200 David Hildenbrand <[email protected]> wrote:
This series cleans up and fixes
break_ksm().
Quite a lot of fixups were needed merging this. I guess you couldn't develop against mm-unstable because the v1 series was already in there.
For this reason I'll henceforth be more inclined to drop serieses when I know a full resend is coming out.
So please do let me know when a full resend is coming out. Or, of course, send little fixes against the current version.
Quoted 6 lines · Andrew Morton
Quoted 1 line · On Fri, 21 Oct 2022 12:11:32 +0200 David Hildenbrand <[email protected]> wrote:
This series cleans up and fixes
break_ksm().Quite a lot of fixups were needed merging this. I guess you couldn't develop against mm-unstable because the v1 series was already in there.
Nowadays, I tend to send against mm-stable (I remember that was the suggestion). Usually it works because there are no conflicts -- this time there are probably quite some kvm unit test conflicts.
Feel free to ask me next time to rebase on XYZ so I can make you life easier ;)
Quoted 3 lines
For this reason I'll henceforth be more inclined to drop serieses when I know a full resend is coming out.
Yes, good idea. While the fixup-patch process works for small adjustments, it's not a good fit for bigger changes, especially once involving new patches.
Quoted 2 lines
So please do let me know when a full resend is coming out.
Will do; I kind-of did that [1] but I should have been more clear ("Please drop the current series, I'll send a new version.").
Quoted 2 lines
Or, of
course, send little fixes against the current version.
I prefer a full resend when there are bigger changes that involve modifications to multiple patches ... which also makes life easier for reviewers.
[1] https://lore.kernel.org/all/[email protected]/T/#u
Signature · 3 lines
Thanks, David / dhildenb