userfaultfd: write protection support
Subsystems (16)
Peter Xu
reviewerv6 cover userfaultfd: write protection support
This v6 series implements initial write protection support for userfaultfd (heavily based on work from Andrea Arcangeli and Shaohua Li for the initial versions, and mostly all the follow up ideas from Andrea too). Currently both shmem and hugetlbfs are not supported yet, but only anonymous memory.
It's based on the fault retry series:
https://lore.kernel.org/lkml/[email protected]/
This series can also be found at (with the mm page fault retry series applied as well):
https://github.com/xzpeter/linux/tree/uffd-wp-merged
Previous version of this series is tested by both Marty Mcfadden <[email protected]> and Bobby Powers <[email protected]>.
My sincerely thanks to everyone who helped to move this forward even a bit! Any comment is welcomed.
Thanks,
v6 changelog:
- rebase
- drop patch "userfaultfd: introduce helper vma_find_uffd" because after rebase to 5.6-rc2 we've got find_dst_vma which does exactly the same thing. Use that instead.
v5 changelog:
- rebase
- drop two patches:
"userfaultfd: wp: handle COW properly for uffd-wp"
"mm: introduce do_wp_page_cont()"
instead remove the write bit always when resolving uffd-wp page fault in previous patch ("userfaultfd: wp: apply _PAGE_UFFD_WP bit") then COW will be handled correctly in the PF irq handler [Andrea]
v4 changelog:
- add r-bs
- use kernel-doc format for fault_flag_allow_retry_first [Jerome]
- drop "export wp_page_copy", add new patch to split do_wp_page(), use it in change_pte_range() to replace the wp_page_copy(). [Jerome] (I thought about different ways to do this but I still can't find a 100% good way for all... in this version I still used the do_wp_page_cont naming. We can still discuss this and how we should split do_wp_page)
- make sure uffd-wp will also apply to device private entries which HMM uses [Jerome]
v3 changelog:
- take r-bs
- patch 1: fix typo [Jerome]
- patch 2: use brackets where proper around (flags & VM_FAULT_RETRY) (there're three places to change, not four...) [Jerome]
- patch 4: make sure TRIED is applied correctly on all archs, add more comment to explain the new page fault mechanism [Jerome]
- patch 7: in do_swap_page() remove the two lines to remove FAULT_FLAG_WRITE flag [Jerome]
- patch 10: another brackets change like above, and in mfill_atomic_pte return -EINVAL when detected wp_copy==1 upon shared memories [Jerome]
- patch 12: move _PAGE_CHG_MASK change to patch 8 [Jerome]
- patch 14: wp_page_copy() - fix write bit; change_pte_range() - detect PTE change after COW [Jerome]
- patch 17: remove last paragraph of commit message, no need to drop the two lines in do_swap_page() since they've been directly dropped in patch 7; touch up remove_migration_pte() to only detect uffd-wp bit if it's read migration entry [Jerome]
- add patch: "userfaultfd: wp: declare _UFFDIO_WRITEPROTECT conditionally", which remove _UFFDIO_WRITEPROTECT bit if detected non-anonymous memory during REGISTER; meanwhile fixup the test case for shmem too for expected ioctls returned from REGISTER [Mike]
- add patch: "userfaultfd: wp: fixup swap entries in change_pte_range", the new patch will allow to apply the uffd-wp bits upon swap entries directly (e.g., when the page is during migration or the page was swapped out). Please see the patch for detail information.
v2 changelog:
- add some r-bs
- split the patch "mm: userfault: return VM_FAULT_RETRY on signals" into two: one to focus on the signal behavior change, the other to remove the NOPAGE special path in handle_userfault(). Removing the ARC specific change and remove that part of commit message since it's fixed in 4d447455e73b already [Jerome]
- return -ENOENT when VMA is invalid for UFFDIO_WRITEPROTECT to match UFFDIO_COPY errno [Mike]
- add a new patch to introduce helper to find valid VMA for uffd [Mike]
- check against VM_MAYWRITE instead of VM_WRITE when registering UFFD WP [Mike]
- MM_CP_DIRTY_ACCT is used incorrectly, fix it up [Jerome]
- make sure the lock_page behavior will not be changed [Jerome]
- reorder the whole series, introduce the new ioctl last. [Jerome]
- fix up the uffdio_writeprotect() following commit df2cc96e77011cf79 to return -EAGAIN when detected mm layout changes [Mike]
v1 can be found at: https://lkml.org/lkml/2019/1/21/130
Any comment would be greatly welcomed. Thanks.
Overview
====================
The uffd-wp work was initialized by Shaohua Li [1], and later continued by Andrea [2]. This series is based upon Andrea's latest userfaultfd tree, and it is a continuous works from both Shaohua and Andrea. Many of the follow up ideas come from Andrea too.
Besides the old MISSING register mode of userfaultfd, the new uffd-wp support provides another alternative register mode called UFFDIO_REGISTER_MODE_WP that can be used to listen to not only missing page faults but also write protection page faults, or even they can be registered together. At the same time, the new feature also provides a new userfaultfd ioctl called UFFDIO_WRITEPROTECT which allows the userspace to write protect a range or memory or fixup write permission of faulted pages.
Please refer to the document patch "userfaultfd: wp:UFFDIO_REGISTER_MODE_WP documentation update" for more information on the new interface and what it can do.
The major workflow of an uffd-wp program should be:
1. Register a memory region with WP mode using UFFDIO_REGISTER_MODE_WP
2. Write protect part of the whole registered region using UFFDIO_WRITEPROTECT, passing in UFFDIO_WRITEPROTECT_MODE_WP to show that we want to write protect the range.
3. Start a working thread that modifies the protected pages, meanwhile listening to UFFD messages.
4. When a write is detected upon the protected range, page fault happens, a UFFD message will be generated and reported to the page fault handling thread
5. The page fault handler thread resolves the page fault using the new UFFDIO_WRITEPROTECT ioctl, but this time passing in !UFFDIO_WRITEPROTECT_MODE_WP instead showing that we want to recover the write permission. Before this operation, the fault handler thread can do anything it wants, e.g., dumps the page to a persistent storage.
6. The worker thread will continue running with the correctly applied write permission from step 5.
Currently there are already two projects that are based on this new userfaultfd feature.
QEMU Live Snapshot: The project provides a way to allow the QEMU
hypervisor to take snapshot of VMs without stopping the VM [3].
LLNL umap library: The project provides a mmap-like interface and
"allow to have an application specific buffer of pages cached from a large file, i.e. out-of-core execution using memory map" [4][5].
Before posting the patchset, this series was smoke tested against QEMU live snapshot and the LLNL umap library (by doing parallel quicksort using 128 sorting threads + 80 uffd servicing threads). My sincere thanks to Marty Mcfadden and Denis Plotnikov for the help along the way.
TODO
=============
- hugetlbfs/shmem support
- performance
- more architectures
- cooperate with mprotect()-allowed processes (???)
- ...
References
==========
[1] https://lwn.net/Articles/666187/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git/log/?h=userfault
[3] https://github.com/denis-plotnikov/qemu/commits/background-snapshot-kvm
[4] https://github.com/LLNL/umap
[5] https://llnl-umap.readthedocs.io/en/develop/
[6] https://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git/commit/?h=userfault&id=b245ecf6cf59156966f3da6e6b674f6695a5ffa5
[7] https://lkml.org/lkml/2018/11/21/370
[8] https://lkml.org/lkml/2018/12/30/64
Andrea Arcangeli (5):
userfaultfd: wp: hook userfault handler to write protection fault
userfaultfd: wp: add WP pagetable tracking to x86
userfaultfd: wp: userfaultfd_pte/huge_pmd_wp() helpers
userfaultfd: wp: add UFFDIO_COPY_MODE_WP
userfaultfd: wp: add the writeprotect API to userfaultfd ioctl
Martin Cracauer (1):
userfaultfd: wp: UFFDIO_REGISTER_MODE_WP documentation update
Peter Xu (10):
mm: merge parameters for change_protection()
userfaultfd: wp: apply _PAGE_UFFD_WP bit
userfaultfd: wp: drop _PAGE_UFFD_WP properly when fork
userfaultfd: wp: add pmd_swp_*uffd_wp() helpers
userfaultfd: wp: support swap and page migration
khugepaged: skip collapse if uffd-wp detected
userfaultfd: wp: don't wake up when doing write protect
userfaultfd: wp: declare _UFFDIO_WRITEPROTECT conditionally
userfaultfd: selftests: refactor statistics
userfaultfd: selftests: add write-protect test
Shaohua Li (3):
userfaultfd: wp: add helper for writeprotect check
userfaultfd: wp: support write protection for userfault vma range
userfaultfd: wp: enabled write protection in userfaultfd API
Documentation/admin-guide/mm/userfaultfd.rst | 51 +++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 67 ++++++ arch/x86/include/asm/pgtable_64.h | 8 +- arch/x86/include/asm/pgtable_types.h | 11 +- fs/userfaultfd.c | 106 +++++++-- include/asm-generic/pgtable.h | 1 + include/asm-generic/pgtable_uffd.h | 66 ++++++ include/linux/huge_mm.h | 2 +- include/linux/mm.h | 19 +- include/linux/swapops.h | 2 + include/linux/userfaultfd_k.h | 42 +++- include/trace/events/huge_memory.h | 1 + include/uapi/linux/userfaultfd.h | 40 +++- init/Kconfig | 5 + mm/huge_memory.c | 32 ++- mm/khugepaged.c | 23 ++ mm/memory.c | 26 ++- mm/mempolicy.c | 2 +- mm/migrate.c | 6 + mm/mprotect.c | 74 ++++-- mm/rmap.c | 6 + mm/userfaultfd.c | 94 +++++++- tools/testing/selftests/vm/userfaultfd.c | 225 +++++++++++++++---- 24 files changed, 791 insertions(+), 119 deletions(-) create mode 100644 include/asm-generic/pgtable_uffd.h
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 1/19 userfaultfd: wp: add helper for writeprotect check
From: Shaohua Li <[email protected]>
add helper for writeprotect check. Will use it later.
--- include/linux/userfaultfd_k.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
include/linux/userfaultfd_k.h +10 -0
@@ -52,6 +52,11 @@ static inline bool userfaultfd_missing(struct vm_area_struct *vma)return vma->vm_flags &VM_UFFD_MISSING ; } +static inline bool userfaultfd_wp (struct vm_area_struct *vma) +{ +return vma->vm_flags &VM_UFFD_WP ; +} +static inline bool userfaultfd_armed (struct vm_area_struct *vma) {return vma->vm_flags & (VM_UFFD_MISSING |VM_UFFD_WP );
@@ -96,6 +101,11 @@ static inline bool userfaultfd_missing(struct vm_area_struct *vma)return false; } +static inline bool userfaultfd_wp (struct vm_area_struct *vma) +{ +return false; +} +static inline bool userfaultfd_armed (struct vm_area_struct *vma) {return false;
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 2/19 userfaultfd: wp: hook userfault handler to write protection fault
In reply to Peter XuFrom: Andrea Arcangeli <[email protected]>
There are several cases write protection fault happens. It could be a write to zero page, swaped page or userfault write protected page. When the fault happens, there is no way to know if userfault write protect the page before. Here we just blindly issue a userfault notification for vma with VM_UFFD_WP regardless if app write protects it yet. Application should be ready to handle such wp fault.
v1: From: Shaohua Li <[email protected]>
v2: Handle the userfault in the common do_wp_page. If we get there a pagetable is present and readonly so no need to do further processing until we solve the userfault.
In the swapin case, always swapin as readonly. This will cause false positive userfaults. We need to decide later if to eliminate them with a flag like soft-dirty in the swap entry (see _PAGE_SWP_SOFT_DIRTY).
hugetlbfs wouldn't need to worry about swapouts but and tmpfs would be handled by a swap entry bit like anonymous memory.
The main problem with no easy solution to eliminate the false positives, will be if/when userfaultfd is extended to real filesystem pagecache. When the pagecache is freed by reclaim we can't leave the radix tree pinned if the inode and in turn the radix tree is reclaimed as well.
The estimation is that full accuracy and lack of false positives could be easily provided only to anonymous memory (as long as there's no fork or as long as MADV_DONTFORK is used on the userfaultfd anonymous range) tmpfs and hugetlbfs, it's most certainly worth to achieve it but in a later incremental patch.
v3: Add hooking point for THP wrprotect faults.
--- mm/memory.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
mm/memory.c +9 -1
@@ -2733,6 +2733,11 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma; +if (userfaultfd_wp (vma)) { +pte_unmap_unlock (vmf->pte, vmf->ptl); +return handle_userfault (vmf,VM_UFFD_WP ); + } + vmf->page =vm_normal_page (vma, vmf->address, vmf->orig_pte);if (!vmf->page) {/*
@@ -3930,8 +3935,11 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf)/* `inline' is required to avoid gcc 4.1.2 build error */ static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf,pmd_t orig_pmd) { -if (vma_is_anonymous (vmf->vma)) +if (vma_is_anonymous (vmf->vma)) { +if (userfaultfd_wp (vmf->vma)) +return handle_userfault (vmf,VM_UFFD_WP );return do_huge_pmd_wp_page (vmf, orig_pmd); + }if (vmf->vma->vm_ops->huge_fault)return vmf->vma->vm_ops->huge_fault (vmf,PE_SIZE_PMD );
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 3/19 userfaultfd: wp: add WP pagetable tracking to x86
In reply to Peter XuFrom: Andrea Arcangeli <[email protected]>
Accurate userfaultfd WP tracking is possible by tracking exactly which virtual memory ranges were writeprotected by userland. We can't relay only on the RW bit of the mapped pagetable because that information is destroyed by fork() or KSM or swap. If we were to relay on that, we'd need to stay on the safe side and generate false positive wp faults for every swapped out page.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 52 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_64.h | 8 ++++- arch/x86/include/asm/pgtable_types.h | 11 +++++- include/asm-generic/pgtable.h | 1 + include/asm-generic/pgtable_uffd.h | 51 +++++++++++++++++++++++++++ init/Kconfig | 5 +++ 7 files changed, 127 insertions(+), 2 deletions(-) create mode 100644 include/asm-generic/pgtable_uffd.h
arch/x86/Kconfig +1 -0
@@ -230,6 +230,7 @@ config X86 select VIRT_TO_BUS select X86_FEATURE_NAMES if PROC_FS select PROC_PID_ARCH_STATUS if PROC_FS + select HAVE_ARCH_USERFAULTFD_WP if USERFAULTFD config INSTRUCTION_DECODER def_bool y
arch/x86/include/asm/pgtable.h +52 -0
76 lines not on this page · raw message
arch/x86/include/asm/pgtable_64.h +7 -1
@@ -189,7 +189,7 @@ extern void sync_global_pgds(unsigned long start, unsigned long end);* * | ... | 11| 10| 9|8|7|6|5| 4| 3|2| 1|0| <- bit number * | ... |SW3|SW2|SW1|G|L|D|A|CD|WT|U| W|P| <- bit names -* | TYPE (59-63) | ~OFFSET (9-58) |0|0|X|X| X| X|X|SD|0| <- swp entry +* | TYPE (59-63) | ~OFFSET (9-58) |0|0|X|X| X| X|F|SD|0| <- swp entry * * G (8) is aliased and used as a PROT_NONE indicator for * !present ptes. We need to start storing swap entries above
@@ -197,9 +197,15 @@ extern void sync_global_pgds(unsigned long start, unsigned long end);* erratum where they can be incorrectly set by hardware on * non-present PTEs. * +* SD Bits 1-4 are not used in non-present format and available for +* special use described below: +* * SD (1) in swp entry is used to store soft dirty bit, which helps us * remember soft dirty over page migration * +* F (2) in swp entry is used to record when a pagetable is +* writeprotected by userfaultfd WP support. +* * Bit 7 in swp entry should be 0 because pmd_present checks not only P, * but also L and G. *
arch/x86/include/asm/pgtable_types.h +10 -1
@@ -32,6 +32,7 @@#define _PAGE_BIT_SPECIAL _PAGE_BIT_SOFTW1#define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1 +#define _PAGE_BIT_UFFD_WP _PAGE_BIT_SOFTW2/* userfaultfd wrprotected */ #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3/* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4
@@ -100,6 +101,14 @@#define _PAGE_SWP_SOFT_DIRTY (_AT(pteval_t, 0))#endif +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP +#define _PAGE_UFFD_WP (_AT(pteval_t, 1) << _PAGE_BIT_UFFD_WP) +#define _PAGE_SWP_UFFD_WP _PAGE_USER +#else +#define _PAGE_UFFD_WP (_AT(pteval_t, 0)) +#define _PAGE_SWP_UFFD_WP (_AT(pteval_t, 0)) +#endif +#if defined(CONFIG_X86_64 ) || defined(CONFIG_X86_PAE )#define _PAGE_NX (_AT(pteval_t, 1) << _PAGE_BIT_NX)#define _PAGE_DEVMAP (_AT(u64, 1) << _PAGE_BIT_DEVMAP)
@@ -118,7 +127,7 @@*/ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ - _PAGE_SOFT_DIRTY | _PAGE_DEVMAP) + _PAGE_SOFT_DIRTY | _PAGE_DEVMAP | _PAGE_UFFD_WP)#define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE)/*
include/asm-generic/pgtable.h +1 -0
@@ -10,6 +10,7 @@#include <linux/mm_types.h> #include <linux/bug.h> #include <linux/errno.h> +#include <asm-generic/pgtable_uffd.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS
include/asm-generic/pgtable_uffd.h new file +51 -0
51 lines not on this page · raw message
init/Kconfig +5 -0
@@ -1554,6 +1554,11 @@ config ADVISE_SYSCALLS applications use these syscalls, you can disable this option to save space. +config HAVE_ARCH_USERFAULTFD_WP + bool + help + Arch has userfaultfd write protection support + config MEMBARRIER bool "Enable membarrier() system call" if EXPERT default y
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 4/19 userfaultfd: wp: userfaultfd_pte/huge_pmd_wp() helpers
In reply to Peter XuFrom: Andrea Arcangeli <[email protected]>
Implement helpers methods to invoke userfaultfd wp faults more
selectively: not only when a wp fault triggers on a vma with vma->vm_flags VM_UFFD_WP set, but only if the _PAGE_UFFD_WP bit is set in the pagetable too.
--- include/linux/userfaultfd_k.h | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
include/linux/userfaultfd_k.h +27 -0
@@ -14,6 +14,8 @@#include <linux/userfaultfd.h> /* linux/include/uapi/linux/userfaultfd.h */ #include <linux/fcntl.h> +#include <linux/mm.h> +#include <asm-generic/pgtable_uffd.h> /* * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining
@@ -57,6 +59,18 @@ static inline bool userfaultfd_wp(struct vm_area_struct *vma)return vma->vm_flags &VM_UFFD_WP ; } +static inline bool userfaultfd_pte_wp (struct vm_area_struct *vma, +pte_t pte) +{ +return userfaultfd_wp (vma) &&pte_uffd_wp (pte); +} + +static inline bool userfaultfd_huge_pmd_wp (struct vm_area_struct *vma, +pmd_t pmd) +{ +return userfaultfd_wp (vma) &&pmd_uffd_wp (pmd); +} +static inline bool userfaultfd_armed (struct vm_area_struct *vma) {return vma->vm_flags & (VM_UFFD_MISSING |VM_UFFD_WP );
@@ -106,6 +120,19 @@ static inline bool userfaultfd_wp(struct vm_area_struct *vma)return false; } +static inline bool userfaultfd_pte_wp (struct vm_area_struct *vma, +pte_t pte) +{ +return false; +} + +static inline bool userfaultfd_huge_pmd_wp (struct vm_area_struct *vma, +pmd_t pmd) +{ +return false; +} + +static inline bool userfaultfd_armed (struct vm_area_struct *vma) {return false;
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 5/19 userfaultfd: wp: add UFFDIO_COPY_MODE_WP
In reply to Peter XuFrom: Andrea Arcangeli <[email protected]>
This allows UFFDIO_COPY to map pages write-protected.
[peterx: switch to VM_WARN_ON_ONCE in mfill_atomic_pte; add brackets
around "dst_vma->vm_flags & VM_WRITE"; fix wordings in comments and commit messages]
--- fs/userfaultfd.c | 5 +++-- include/linux/userfaultfd_k.h | 2 +- include/uapi/linux/userfaultfd.h | 11 +++++----- mm/userfaultfd.c | 36 ++++++++++++++++++++++---------- 4 files changed, 35 insertions(+), 19 deletions(-)
fs/userfaultfd.c +3 -2
@@ -1724,11 +1724,12 @@ static int userfaultfd_copy(struct userfaultfd_ctx *ctx, ret = -EINVAL ;if (uffdio_copy.src + uffdio_copy.len <= uffdio_copy.src) goto out; -if (uffdio_copy.mode & ~UFFDIO_COPY_MODE_DONTWAKE ) +if (uffdio_copy.mode & ~(UFFDIO_COPY_MODE_DONTWAKE |UFFDIO_COPY_MODE_WP )) goto out;if (mmget_not_zero (ctx->mm)) { ret =mcopy_atomic (ctx->mm, uffdio_copy.dst, uffdio_copy.src, - uffdio_copy.len, &ctx->mmap_changing); + uffdio_copy.len, &ctx->mmap_changing, + uffdio_copy.mode);mmput (ctx->mm); }else {return -ESRCH ;
include/linux/userfaultfd_k.h +1 -1
@@ -36,7 +36,7 @@ extern vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason);extern ssize_t mcopy_atomic (struct mm_struct *dst_mm,unsigned long dst_start,unsigned long src_start,unsigned long len, -bool *mmap_changing); +bool *mmap_changing,__u64 mode); extern ssize_t mfill_zeropage(struct mm_struct *dst_mm, unsigned long dst_start, unsigned long len,
include/uapi/linux/userfaultfd.h +6 -5
@@ -203,13 +203,14 @@ struct uffdio_copy {__u64 dst;__u64 src;__u64 len; +#define UFFDIO_COPY_MODE_DONTWAKE ((__u64)1<<0)/* -* There will be a wrprotection flag later that allows to map -* pages wrprotected on the fly. And such a flag will be -* available if the wrprotection ioctl are implemented for the -* range according to the uffdio_register.ioctls. +* UFFDIO_COPY_MODE_WP will map the page write protected on +* the fly. UFFDIO_COPY_MODE_WP is available only if the +* write protected ioctl is implemented for the range +* according to the uffdio_register.ioctls. */ -#define UFFDIO_COPY_MODE_DONTWAKE ((__u64)1<<0) +#define UFFDIO_COPY_MODE_WP ((__u64)1<<1)__u64 mode;/*
mm/userfaultfd.c +25 -11
@@ -53,7 +53,8 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - struct page **pagep) + struct page **pagep, + bool wp_copy) { struct mem_cgroup *memcg;pte_t _dst_pte, *dst_pte;
@@ -99,9 +100,9 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,if (mem_cgroup_try_charge (page, dst_mm,GFP_KERNEL , &memcg, false)) goto out_release; - _dst_pte =mk_pte (page, dst_vma->vm_page_prot); -if (dst_vma->vm_flags &VM_WRITE ) - _dst_pte =pte_mkwrite (pte_mkdirty (_dst_pte)); + _dst_pte =pte_mkdirty (mk_pte (page, dst_vma->vm_page_prot)); +if ((dst_vma->vm_flags &VM_WRITE ) && !wp_copy) + _dst_pte =pte_mkwrite (_dst_pte); dst_pte =pte_offset_map_lock (dst_mm, dst_pmd, dst_addr, &ptl);if (dst_vma->vm_file) {
@@ -408,7 +409,8 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm,unsigned long dst_addr, unsigned long src_addr, struct page **page, - bool zeropage) + bool zeropage, + bool wp_copy) { ssize_t err;
@@ -425,11 +427,13 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm,if (!(dst_vma->vm_flags &VM_SHARED )) {if (!zeropage) err =mcopy_atomic_pte (dst_mm, dst_pmd, dst_vma, - dst_addr, src_addr, page); + dst_addr, src_addr, page, + wp_copy);else err =mfill_zeropage_pte (dst_mm, dst_pmd, dst_vma, dst_addr); }else { +VM_WARN_ON_ONCE (wp_copy);if (!zeropage) err = shmem_mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr,
@@ -447,7 +451,8 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,unsigned long src_start, unsigned long len, bool zeropage, - bool *mmap_changing) + bool *mmap_changing, + __u64 mode) { struct vm_area_struct *dst_vma;ssize_t err;
@@ -455,6 +460,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,unsigned long src_addr, dst_addr;long copied;struct page *page; +bool wp_copy;/* * Sanitize the command parameters:
@@ -500,6 +506,14 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, dst_vma->vm_flags & VM_SHARED)) goto out_unlock; +/* +* validate 'mode' now that we know the dst_vma: don't allow +* a wrprotect copy if the userfaultfd didn't register as WP. +*/ + wp_copy = mode &UFFDIO_COPY_MODE_WP ; +if (wp_copy && !(dst_vma->vm_flags &VM_UFFD_WP )) + goto out_unlock; +/* * If this is a HUGETLB vma, pass off to appropriate routine */
@@ -555,7 +569,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,BUG_ON (pmd_trans_huge (*dst_pmd)); err =mfill_atomic_pte (dst_mm, dst_pmd, dst_vma, dst_addr, - src_addr, &page, zeropage); + src_addr, &page, zeropage, wp_copy);cond_resched ();if (unlikely (err == -ENOENT )) {
@@ -602,14 +616,14 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,ssize_t mcopy_atomic (struct mm_struct *dst_mm,unsigned long dst_start,unsigned long src_start,unsigned long len, -bool *mmap_changing) +bool *mmap_changing,__u64 mode) {return __mcopy_atomic (dst_mm, dst_start, src_start, len, false, - mmap_changing); + mmap_changing, mode); }ssize_t mfill_zeropage (struct mm_struct *dst_mm,unsigned long start,unsigned long len,bool *mmap_changing) { -return __mcopy_atomic (dst_mm, start,0 , len, true, mmap_changing); +return __mcopy_atomic (dst_mm, start,0 , len, true, mmap_changing,0 ); }
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 6/19 mm: merge parameters for change_protection()
In reply to Peter Xuchange_protection() was used by either the NUMA or mprotect() code, there's one parameter for each of the callers (dirty_accountable and prot_numa). Further, these parameters are passed along the calls:
- change_protection_range()
- change_p4d_range()
- change_pud_range()
- change_pmd_range()
- ...
Now we introduce a flag for change_protect() and all these helpers to replace these parameters. Then we can avoid passing multiple parameters multiple times along the way.
More importantly, it'll greatly simplify the work if we want to introduce any new parameters to change_protection(). In the follow up patches, a new parameter for userfaultfd write protection will be introduced.
No functional change at all.
--- include/linux/huge_mm.h | 2 +- include/linux/mm.h | 14 +++++++++++++- mm/huge_memory.c | 3 ++- mm/mempolicy.c | 2 +- mm/mprotect.c | 29 ++++++++++++++++------------- 5 files changed, 33 insertions(+), 17 deletions(-)
include/linux/huge_mm.h +1 -1
@@ -46,7 +46,7 @@ extern bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, pmd_t *old_pmd, pmd_t *new_pmd);extern int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot, -int prot_numa); +unsigned long cp_flags);vm_fault_t vmf_insert_pfn_pmd (struct vm_fault *vmf,pfn_t pfn,bool write);vm_fault_t vmf_insert_pfn_pud (struct vm_fault *vmf,pfn_t pfn,bool write);enum transparent_hugepage_flag {
include/linux/mm.h +13 -1
@@ -1660,9 +1660,21 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma, unsigned long old_addr, struct vm_area_struct *new_vma, unsigned long new_addr, unsigned long len, bool need_rmap_locks); + +/* +* Flags used by change_protection(). For now we make it a bitmap so +* that we can pass in multiple flags just like parameters. However +* for now all the callers are only use one of the flags at the same +* time. +*/ +/* Whether we should allow dirty bit accounting */ +#define MM_CP_DIRTY_ACCT (1UL << 0) +/* Whether this protection change is for NUMA hints */ +#define MM_CP_PROT_NUMA (1UL << 1) +extern unsigned long change_protection (struct vm_area_struct *vma,unsigned long start,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa); +unsigned long cp_flags);extern int mprotect_fixup (struct vm_area_struct *vma,struct vm_area_struct **pprev,unsigned long start,unsigned long end,unsigned long newflags);
mm/huge_memory.c +2 -1
@@ -1934,13 +1934,14 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,* - HPAGE_PMD_NR is protections changed and TLB flush necessary */ int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long addr,pgprot_t newprot,int prot_numa) +unsigned long addr,pgprot_t newprot,unsigned long cp_flags) {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl;pmd_t entry;bool preserve_write;int ret; +bool prot_numa = cp_flags &MM_CP_PROT_NUMA ; ptl =__pmd_trans_huge_lock (pmd, vma);if (!ptl)
mm/mempolicy.c +1 -1
@@ -598,7 +598,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, {int nr_updated; - nr_updated =change_protection (vma, addr, end,PAGE_NONE ,0 ,1 ); + nr_updated =change_protection (vma, addr, end,PAGE_NONE ,MM_CP_PROT_NUMA );if (nr_updated)count_vm_numa_events (NUMA_PTE_UPDATES , nr_updated);
mm/mprotect.c +16 -13
@@ -37,12 +37,14 @@static unsigned long change_pte_range (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa) +unsigned long cp_flags) {pte_t *pte, oldpte;spinlock_t *ptl;unsigned long pages =0 ;int target_node =NUMA_NO_NODE ; +bool dirty_accountable = cp_flags &MM_CP_DIRTY_ACCT ; +bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;/* * Can be called with only the mmap_sem for reading by
@@ -163,7 +165,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,static inline unsigned long change_pmd_range (struct vm_area_struct *vma,pud_t *pud,unsigned long addr,unsigned long end, -pgprot_t newprot,int dirty_accountable,int prot_numa) +pgprot_t newprot,unsigned long cp_flags) {pmd_t *pmd;unsigned long next;
@@ -195,7 +197,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,__split_huge_pmd (vma, pmd, addr, false,NULL ); }else {int nr_ptes =change_huge_pmd (vma, pmd, addr, - newprot, prot_numa); + newprot, cp_flags);if (nr_ptes) {if (nr_ptes ==HPAGE_PMD_NR ) {
@@ -210,7 +212,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,/* fall through, the trans huge pmd just split */ } this_pages =change_pte_range (vma, pmd, addr, next, newprot, - dirty_accountable, prot_numa); + cp_flags); pages += this_pages; next:cond_resched ();
@@ -226,7 +228,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,static inline unsigned long change_pud_range (struct vm_area_struct *vma,p4d_t *p4d,unsigned long addr,unsigned long end, -pgprot_t newprot,int dirty_accountable,int prot_numa) +pgprot_t newprot,unsigned long cp_flags) {pud_t *pud;unsigned long next;
@@ -238,7 +240,7 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma,if (pud_none_or_clear_bad (pud))continue ; pages +=change_pmd_range (vma, pud, addr, next, newprot, - dirty_accountable, prot_numa); + cp_flags); }while (pud++, addr = next, addr != end);return pages;
@@ -246,7 +248,7 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma,static inline unsigned long change_p4d_range (struct vm_area_struct *vma,pgd_t *pgd,unsigned long addr,unsigned long end, -pgprot_t newprot,int dirty_accountable,int prot_numa) +pgprot_t newprot,unsigned long cp_flags) {p4d_t *p4d;unsigned long next;
@@ -258,7 +260,7 @@ static inline unsigned long change_p4d_range(struct vm_area_struct *vma,if (p4d_none_or_clear_bad (p4d))continue ; pages +=change_pud_range (vma, p4d, addr, next, newprot, - dirty_accountable, prot_numa); + cp_flags); }while (p4d++, addr = next, addr != end);return pages;
@@ -266,7 +268,7 @@ static inline unsigned long change_p4d_range(struct vm_area_struct *vma,static unsigned long change_protection_range (struct vm_area_struct *vma,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa) +unsigned long cp_flags) {struct mm_struct *mm = vma->vm_mm;pgd_t *pgd;
@@ -283,7 +285,7 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,if (pgd_none_or_clear_bad (pgd))continue ; pages +=change_p4d_range (vma, pgd, addr, next, newprot, - dirty_accountable, prot_numa); + cp_flags); }while (pgd++, addr = next, addr != end);/* Only flush the TLB if we actually modified any entries: */
@@ -296,14 +298,15 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,unsigned long change_protection (struct vm_area_struct *vma,unsigned long start,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa) +unsigned long cp_flags) {unsigned long pages;if (is_vm_hugetlb_page (vma)) pages =hugetlb_change_protection (vma, start, end, newprot);else - pages =change_protection_range (vma, start, end, newprot, dirty_accountable, prot_numa); + pages =change_protection_range (vma, start, end, newprot, + cp_flags);return pages; }
@@ -425,7 +428,7 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,vma_set_page_prot (vma);change_protection (vma, start, end, vma->vm_page_prot, - dirty_accountable,0 ); + dirty_accountable ?MM_CP_DIRTY_ACCT :0 );/* * Private VM_LOCKED VMA becoming writable: trigger COW to avoid major
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 7/19 userfaultfd: wp: apply _PAGE_UFFD_WP bit
In reply to Peter XuFirstly, introduce two new flags MM_CP_UFFD_WP[_RESOLVE] for change_protection() when used with uffd-wp and make sure the two new flags are exclusively used. Then,
- For MM_CP_UFFD_WP: apply the _PAGE_UFFD_WP bit and remove _PAGE_RW when a range of memory is write protected by uffd
- For MM_CP_UFFD_WP_RESOLVE: remove the _PAGE_UFFD_WP bit and recover _PAGE_RW when write protection is resolved from userspace
And use this new interface in mwriteprotect_range() to replace the old MM_CP_DIRTY_ACCT.
Do this change for both PTEs and huge PMDs. Then we can start to identify which PTE/PMD is write protected by general (e.g., COW or soft dirty tracking), and which is for userfaultfd-wp.
Since we should keep the _PAGE_UFFD_WP when doing pte_modify(), add it into _PAGE_CHG_MASK as well. Meanwhile, since we have this new bit, we can be even more strict when detecting uffd-wp page faults in either do_wp_page() or wp_huge_pmd().
After we're with _PAGE_UFFD_WP, a special case is when a page is both protected by the general COW logic and also userfault-wp. Here the userfault-wp will have higher priority and will be handled first. Only after the uffd-wp bit is cleared on the PTE/PMD will we continue to handle the general COW. These are the steps on what will happen with such a page:
1. CPU accesses write protected shared page (so both protected by general COW and uffd-wp), blocked by uffd-wp first because in do_wp_page we'll handle uffd-wp first, so it has higher priority than general COW.
2. Uffd service thread receives the request, do UFFDIO_WRITEPROTECT to remove the uffd-wp bit upon the PTE/PMD. However here we still keep the write bit cleared. Notify the blocked CPU.
3. The blocked CPU resumes the page fault process with a fault retry, during retry it'll notice it was not with the uffd-wp bit this time but it is still write protected by general COW, then it'll go though the COW path in the fault handler, copy the page, apply write bit where necessary, and retry again.
4. The CPU will be able to access this page with write bit set.
--- include/linux/mm.h | 5 +++++ mm/huge_memory.c | 18 +++++++++++++++++- mm/memory.c | 4 ++-- mm/mprotect.c | 17 +++++++++++++++++ mm/userfaultfd.c | 8 ++++++-- 5 files changed, 47 insertions(+), 5 deletions(-)
include/linux/mm.h +5 -0
@@ -1671,6 +1671,11 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,#define MM_CP_DIRTY_ACCT (1UL << 0)/* Whether this protection change is for NUMA hints */ #define MM_CP_PROT_NUMA (1UL << 1) +/* Whether this change is for write protecting */ +#define MM_CP_UFFD_WP (1UL << 2)/* do wp */ +#define MM_CP_UFFD_WP_RESOLVE (1UL << 3)/* Resolve wp */ +#define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ + MM_CP_UFFD_WP_RESOLVE) extern unsigned long change_protection(struct vm_area_struct *vma, unsigned long start, unsigned long end, pgprot_t newprot,
mm/huge_memory.c +17 -1
@@ -1942,6 +1942,8 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,bool preserve_write;int ret;bool prot_numa = cp_flags &MM_CP_PROT_NUMA ; +bool uffd_wp = cp_flags &MM_CP_UFFD_WP ; +bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; ptl =__pmd_trans_huge_lock (pmd, vma);if (!ptl)
@@ -2008,6 +2010,17 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, entry =pmd_modify (entry, newprot);if (preserve_write) entry =pmd_mk_savedwrite (entry); +if (uffd_wp) { + entry =pmd_wrprotect (entry); + entry =pmd_mkuffd_wp (entry); + }else if (uffd_wp_resolve) { +/* +* Leave the write bit to be handled by PF interrupt +* handler, then things like COW could be properly +* handled. +*/ + entry =pmd_clear_uffd_wp (entry); + } ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry);BUG_ON (vma_is_anonymous (vma) && !preserve_write &&pmd_write (entry));
@@ -2156,7 +2169,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,struct page *page;pgtable_t pgtable;pmd_t old_pmd, _pmd; -bool young, write, soft_dirty, pmd_migration = false; +bool young, write, soft_dirty, pmd_migration = false, uffd_wp = false;unsigned long addr;int i;
@@ -2238,6 +2251,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, write =pmd_write (old_pmd); young =pmd_young (old_pmd); soft_dirty =pmd_soft_dirty (old_pmd); + uffd_wp =pmd_uffd_wp (old_pmd); }VM_BUG_ON_PAGE (!page_count (page), page);page_ref_add (page,HPAGE_PMD_NR -1 );
@@ -2271,6 +2285,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry =pte_mkold (entry);if (soft_dirty) entry =pte_mksoft_dirty (entry); +if (uffd_wp) + entry =pte_mkuffd_wp (entry); } pte =pte_offset_map (&_pmd, addr);BUG_ON (!pte_none (*pte));
mm/memory.c +2 -2
@@ -2733,7 +2733,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma; -if (userfaultfd_wp (vma)) { +if (userfaultfd_pte_wp (vma, *vmf->pte)) {pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_WP ); }
@@ -3936,7 +3936,7 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf)static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf,pmd_t orig_pmd) {if (vma_is_anonymous (vmf->vma)) { -if (userfaultfd_wp (vmf->vma)) +if (userfaultfd_huge_pmd_wp (vmf->vma, orig_pmd))return handle_userfault (vmf,VM_UFFD_WP );return do_huge_pmd_wp_page (vmf, orig_pmd); }
mm/mprotect.c +17 -0
@@ -45,6 +45,8 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,int target_node =NUMA_NO_NODE ;bool dirty_accountable = cp_flags &MM_CP_DIRTY_ACCT ;bool prot_numa = cp_flags &MM_CP_PROT_NUMA ; +bool uffd_wp = cp_flags &MM_CP_UFFD_WP ; +bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ;/* * Can be called with only the mmap_sem for reading by
@@ -116,6 +118,19 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (preserve_write) ptent =pte_mk_savedwrite (ptent); +if (uffd_wp) { + ptent =pte_wrprotect (ptent); + ptent =pte_mkuffd_wp (ptent); + }else if (uffd_wp_resolve) { +/* +* Leave the write bit to be handled +* by PF interrupt handler, then +* things like COW could be properly +* handled. +*/ + ptent =pte_clear_uffd_wp (ptent); + } +/* Avoid taking write faults for known dirty pages */ if (dirty_accountable && pte_dirty(ptent) && (pte_soft_dirty(ptent) ||
@@ -302,6 +317,8 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start, {unsigned long pages; +BUG_ON ((cp_flags &MM_CP_UFFD_WP_ALL ) ==MM_CP_UFFD_WP_ALL ); +if (is_vm_hugetlb_page (vma)) pages =hugetlb_change_protection (vma, start, end, newprot);else
mm/userfaultfd.c +6 -2
@@ -101,8 +101,12 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, goto out_release; _dst_pte =pte_mkdirty (mk_pte (page, dst_vma->vm_page_prot)); -if ((dst_vma->vm_flags &VM_WRITE ) && !wp_copy) - _dst_pte =pte_mkwrite (_dst_pte); +if (dst_vma->vm_flags &VM_WRITE ) { +if (wp_copy) + _dst_pte =pte_mkuffd_wp (_dst_pte); +else + _dst_pte =pte_mkwrite (_dst_pte); + } dst_pte =pte_offset_map_lock (dst_mm, dst_pmd, dst_addr, &ptl);if (dst_vma->vm_file) {
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 8/19 userfaultfd: wp: drop _PAGE_UFFD_WP properly when fork
In reply to Peter XuUFFD_EVENT_FORK support for uffd-wp should be already there, except that we should clean the uffd-wp bit if uffd fork event is not enabled. Detect that to avoid _PAGE_UFFD_WP being set even if the VMA is not being tracked by VM_UFFD_WP. Do this for both small PTEs and huge PMDs.
--- mm/huge_memory.c | 8 ++++++++ mm/memory.c | 8 ++++++++ 2 files changed, 16 insertions(+)
mm/huge_memory.c +8 -0
@@ -1011,6 +1011,14 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN ; pmd = *src_pmd; +/* +* Make sure the _PAGE_UFFD_WP bit is cleared if the new VMA +* does not have the VM_UFFD_WP, which means that the uffd +* fork event is not enabled. +*/ +if (!(vma->vm_flags &VM_UFFD_WP )) + pmd =pmd_clear_uffd_wp (pmd); +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (unlikely (is_swap_pmd (pmd))) {swp_entry_t entry =pmd_to_swp_entry (pmd);
mm/memory.c +8 -0
@@ -785,6 +785,14 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte =pte_mkclean (pte); pte =pte_mkold (pte); +/* +* Make sure the _PAGE_UFFD_WP bit is cleared if the new VMA +* does not have the VM_UFFD_WP, which means that the uffd +* fork event is not enabled. +*/ +if (!(vm_flags &VM_UFFD_WP )) + pte =pte_clear_uffd_wp (pte); + page =vm_normal_page (vma, addr, pte);if (page) {get_page (page);
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 9/19 userfaultfd: wp: add pmd_swp_*uffd_wp() helpers
In reply to Peter XuAdding these missing helpers for uffd-wp operations with pmd swap/migration entries.
--- arch/x86/include/asm/pgtable.h | 15 +++++++++++++++ include/asm-generic/pgtable_uffd.h | 15 +++++++++++++++ 2 files changed, 30 insertions(+)
arch/x86/include/asm/pgtable.h +15 -0
@@ -1427,6 +1427,21 @@ static inline pte_t pte_swp_clear_uffd_wp(pte_t pte) {return pte_clear_flags (pte, _PAGE_SWP_UFFD_WP); } + +static inline pmd_t pmd_swp_mkuffd_wp (pmd_t pmd) +{ +return pmd_set_flags (pmd, _PAGE_SWP_UFFD_WP); +} + +static inline int pmd_swp_uffd_wp (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_SWP_UFFD_WP; +} + +static inline pmd_t pmd_swp_clear_uffd_wp (pmd_t pmd) +{ +return pmd_clear_flags (pmd, _PAGE_SWP_UFFD_WP); +}#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ #define PKRU_AD_BIT 0x1
include/asm-generic/pgtable_uffd.h +15 -0
@@ -46,6 +46,21 @@ static __always_inline pte_t pte_swp_clear_uffd_wp(pte_t pte) {return pte; } + +static inline pmd_t pmd_swp_mkuffd_wp (pmd_t pmd) +{ +return pmd; +} + +static inline int pmd_swp_uffd_wp (pmd_t pmd) +{ +return 0 ; +} + +static inline pmd_t pmd_swp_clear_uffd_wp (pmd_t pmd) +{ +return pmd; +}#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ #endif /* _ASM_GENERIC_PGTABLE_UFFD_H */
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 10/19 userfaultfd: wp: support swap and page migration
In reply to Peter XuFor either swap and page migration, we all use the bit 2 of the entry to identify whether this entry is uffd write-protected. It plays a similar role as the existing soft dirty bit in swap entries but only for keeping the uffd-wp tracking for a specific PTE/PMD.
Something special here is that when we want to recover the uffd-wp bit from a swap/migration entry to the PTE bit we'll also need to take care of the _PAGE_RW bit and make sure it's cleared, otherwise even with the _PAGE_UFFD_WP bit we can't trap it at all.
In change_pte_range() we do nothing for uffd if the PTE is a swap entry. That can lead to data mismatch if the page that we are going to write protect is swapped out when sending the UFFDIO_WRITEPROTECT. This patch also applies/removes the uffd-wp bit even for the swap entries.
--- include/linux/swapops.h | 2 ++ mm/huge_memory.c | 3 +++ mm/memory.c | 8 ++++++++ mm/migrate.c | 6 ++++++ mm/mprotect.c | 28 +++++++++++++++++----------- mm/rmap.c | 6 ++++++ 6 files changed, 42 insertions(+), 11 deletions(-)
include/linux/swapops.h +2 -0
@@ -68,6 +68,8 @@ static inline swp_entry_t pte_to_swp_entry(pte_t pte)if (pte_swp_soft_dirty (pte)) pte =pte_swp_clear_soft_dirty (pte); +if (pte_swp_uffd_wp (pte)) + pte =pte_swp_clear_uffd_wp (pte); arch_entry =__pte_to_swp_entry (pte);return swp_entry (__swp_type (arch_entry),__swp_offset (arch_entry)); }
mm/huge_memory.c +3 -0
@@ -2252,6 +2252,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, write =is_write_migration_entry (entry); young = false; soft_dirty =pmd_swp_soft_dirty (old_pmd); + uffd_wp =pmd_swp_uffd_wp (old_pmd); }else { page =pmd_page (old_pmd);if (pmd_dirty (old_pmd))
@@ -2284,6 +2285,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry =swp_entry_to_pte (swp_entry);if (soft_dirty) entry =pte_swp_mksoft_dirty (entry); +if (uffd_wp) + entry =pte_swp_mkuffd_wp (entry); }else { entry =mk_pte (page + i,READ_ONCE (vma->vm_page_prot)); entry =maybe_mkwrite (entry, vma);
mm/memory.c +8 -0
@@ -733,6 +733,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte =swp_entry_to_pte (entry);if (pte_swp_soft_dirty (*src_pte)) pte =pte_swp_mksoft_dirty (pte); +if (pte_swp_uffd_wp (*src_pte)) + pte =pte_swp_mkuffd_wp (pte);set_pte_at (src_mm, addr, src_pte, pte); } }else if (is_device_private_entry (entry)) {
@@ -762,6 +764,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, is_cow_mapping(vm_flags)) {make_device_private_entry_read (&entry); pte =swp_entry_to_pte (entry); +if (pte_swp_uffd_wp (*src_pte)) + pte =pte_swp_mkuffd_wp (pte);set_pte_at (src_mm, addr, src_pte, pte); } }
@@ -3079,6 +3083,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)flush_icache_page (vma, page);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte); +if (pte_swp_uffd_wp (vmf->orig_pte)) { + pte =pte_mkuffd_wp (pte); + pte =pte_wrprotect (pte); + }set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte);arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); vmf->orig_pte = pte;
mm/migrate.c +6 -0
@@ -243,11 +243,15 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma, entry =pte_to_swp_entry (*pvmw.pte);if (is_write_migration_entry (entry)) pte =maybe_mkwrite (pte, vma); +else if (pte_swp_uffd_wp (*pvmw.pte)) + pte =pte_mkuffd_wp (pte);if (unlikely (is_zone_device_page (new))) {if (is_device_private_page (new)) { entry =make_device_private_entry (new,pte_write (pte)); pte =swp_entry_to_pte (entry); +if (pte_swp_uffd_wp (*pvmw.pte)) + pte =pte_mkuffd_wp (pte); } }
@@ -2318,6 +2322,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, swp_pte =swp_entry_to_pte (entry);if (pte_soft_dirty (pte)) swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pte)) + swp_pte =pte_swp_mkuffd_wp (swp_pte);set_pte_at (mm, addr, ptep, swp_pte);/*
mm/mprotect.c +17 -11
@@ -139,11 +139,11 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); pages++; - }else if (IS_ENABLED (CONFIG_MIGRATION )) { + }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte); +pte_t newpte;if (is_write_migration_entry (entry)) { -pte_t newpte;/* * A protection check is difficult so * just be safe and disable write
@@ -152,22 +152,28 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, newpte =swp_entry_to_pte (entry);if (pte_swp_soft_dirty (oldpte)) newpte =pte_swp_mksoft_dirty (newpte); -set_pte_at (vma->vm_mm, addr, pte, newpte); - - pages++; - } - -if (is_write_device_private_entry (entry)) { -pte_t newpte; - +if (pte_swp_uffd_wp (oldpte)) + newpte =pte_swp_mkuffd_wp (newpte); + }else if (is_write_device_private_entry (entry)) {/* * We do not preserve soft-dirtiness. See * copy_one_pte() for explanation. */ make_device_private_entry_read (&entry); newpte =swp_entry_to_pte (entry); -set_pte_at (vma->vm_mm, addr, pte, newpte); +if (pte_swp_uffd_wp (oldpte)) + newpte =pte_swp_mkuffd_wp (newpte); + }else { + newpte = oldpte; + } +if (uffd_wp) + newpte =pte_swp_mkuffd_wp (newpte); +else if (uffd_wp_resolve) + newpte =pte_swp_clear_uffd_wp (newpte); + +if (!pte_same (oldpte, newpte)) { +set_pte_at (vma->vm_mm, addr, pte, newpte); pages++; } }
mm/rmap.c +6 -0
@@ -1497,6 +1497,8 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, swp_pte =swp_entry_to_pte (entry);if (pte_soft_dirty (pteval)) swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte);set_pte_at (mm, pvmw.address, pvmw.pte, swp_pte);/* * No need to invalidate here it will synchronize on
@@ -1596,6 +1598,8 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, swp_pte =swp_entry_to_pte (entry);if (pte_soft_dirty (pteval)) swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte);set_pte_at (mm, address, pvmw.pte, swp_pte);/* * No need to invalidate here it will synchronize on
@@ -1662,6 +1666,8 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, swp_pte =swp_entry_to_pte (entry);if (pte_soft_dirty (pteval)) swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte);set_pte_at (mm, address, pvmw.pte, swp_pte);/* Invalidate as we cleared the pte */ mmu_notifier_invalidate_range(mm, address,
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 11/19 khugepaged: skip collapse if uffd-wp detected
In reply to Peter XuDon't collapse the huge PMD if there is any userfault write protected small PTEs. The problem is that the write protection is in small page granularity and there's no way to keep all these write protection information if the small pages are going to be merged into a huge PMD.
The same thing needs to be considered for swap entries and migration entries. So do the check as well disregarding khugepaged_max_ptes_swap.
--- include/trace/events/huge_memory.h | 1 + mm/khugepaged.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+)
include/trace/events/huge_memory.h +1 -0
@@ -13,6 +13,7 @@ EM( SCAN_PMD_NULL, "pmd_null") \ EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \ EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \ + EM( SCAN_PTE_UFFD_WP, "pte_uffd_wp") \ EM( SCAN_PAGE_RO, "no_writable_page") \ EM( SCAN_LACK_REFERENCED_PAGE, "lack_referenced_page") \EM (SCAN_PAGE_NULL ,"page_null" ) \
mm/khugepaged.c +23 -0
@@ -29,6 +29,7 @@ enum scan_result {SCAN_PMD_NULL ,SCAN_EXCEED_NONE_PTE ,SCAN_PTE_NON_PRESENT , +SCAN_PTE_UFFD_WP ,SCAN_PAGE_RO ,SCAN_LACK_REFERENCED_PAGE , SCAN_PAGE_NULL,
@@ -1141,6 +1142,15 @@ static int khugepaged_scan_pmd(struct mm_struct *mm,pte_t pteval = *_pte;if (is_swap_pte (pteval)) {if (++unmapped <= khugepaged_max_ptes_swap) { +/* +* Always be strict with uffd-wp +* enabled swap entries. Please see +* comment below for pte_uffd_wp(). +*/ +if (pte_swp_uffd_wp (pteval)) { + result =SCAN_PTE_UFFD_WP ; + goto out_unmap; + }continue ; }else { result =SCAN_EXCEED_SWAP_PTE ;
@@ -1160,6 +1170,19 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, result =SCAN_PTE_NON_PRESENT ; goto out_unmap; } +if (pte_uffd_wp (pteval)) { +/* +* Don't collapse the page if any of the small +* PTEs are armed with uffd write protection. +* Here we can also mark the new huge pmd as +* write protected if any of the small ones is +* marked but that could bring uknown +* userfault messages that falls outside of +* the registered range. So, just be simple. +*/ + result =SCAN_PTE_UFFD_WP ; + goto out_unmap; + }if (pte_write (pteval)) writable = true;
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 12/19 userfaultfd: wp: support write protection for userfault vma range
In reply to Peter XuFrom: Shaohua Li <[email protected]>
Add API to enable/disable writeprotect a vma range. Unlike mprotect, this doesn't split/merge vmas.
[peterx:
- use the helper to find VMA;
- return -ENOENT if not found to match mcopy case;
- use the new MM_CP_UFFD_WP* flags for change_protection
- check against mmap_changing for failures
- replace find_dst_vma with vma_find_uffd]
--- include/linux/userfaultfd_k.h | 3 ++ mm/userfaultfd.c | 54 +++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)
include/linux/userfaultfd_k.h +3 -0
@@ -41,6 +41,9 @@ extern ssize_t mfill_zeropage(struct mm_struct *dst_mm,unsigned long dst_start, unsigned long len, bool *mmap_changing); +extern int mwriteprotect_range (struct mm_struct *dst_mm, +unsigned long start,unsigned long len, +bool enable_wp,bool *mmap_changing);/* mm helpers */ static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,
mm/userfaultfd.c +54 -0
57 lines not on this page · raw message
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 13/19 userfaultfd: wp: add the writeprotect API to userfaultfd ioctl
In reply to Peter XuFrom: Andrea Arcangeli <[email protected]>
v1: From: Shaohua Li <[email protected]>
v2: cleanups, remove a branch.
[peterx writes up the commit message, as below...]
This patch introduces the new uffd-wp APIs for userspace.
Firstly, we'll allow to do UFFDIO_REGISTER with write protection tracking using the new UFFDIO_REGISTER_MODE_WP flag. Note that this flag can co-exist with the existing UFFDIO_REGISTER_MODE_MISSING, in which case the userspace program can not only resolve missing page faults, and at the same time tracking page data changes along the way.
Secondly, we introduced the new UFFDIO_WRITEPROTECT API to do page level write protection tracking. Note that we will need to register the memory region with UFFDIO_REGISTER_MODE_WP before that.
[peterx: remove useless block, write commit message, check against
VM_MAYWRITE rather than VM_WRITE when register]
--- fs/userfaultfd.c | 82 +++++++++++++++++++++++++------- include/uapi/linux/userfaultfd.h | 23 +++++++++ 2 files changed, 89 insertions(+), 16 deletions(-)
fs/userfaultfd.c +66 -16
@@ -314,8 +314,11 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,if (!pmd_present (_pmd)) goto out; -if (pmd_trans_huge (_pmd)) +if (pmd_trans_huge (_pmd)) { +if (!pmd_write (_pmd) && (reason &VM_UFFD_WP )) + ret = true; goto out; + }/* * the pmd is stable (as in !pmd_trans_unstable) so we can re-read it
@@ -328,6 +331,8 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,*/ if (pte_none (*pte)) ret = true; +if (!pte_write (*pte) && (reason &VM_UFFD_WP )) + ret = true;pte_unmap (pte); out:
@@ -1287,10 +1292,13 @@ static __always_inline int validate_range(struct mm_struct *mm,return 0 ; } -static inline bool vma_can_userfault (struct vm_area_struct *vma) +static inline bool vma_can_userfault (struct vm_area_struct *vma, +unsigned long vm_flags) { -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); +/* FIXME: add WP support to hugetlbfs and shmem */ +return vma_is_anonymous (vma) || + ((is_vm_hugetlb_page (vma) ||vma_is_shmem (vma)) && + !(vm_flags &VM_UFFD_WP )); } static int userfaultfd_register(struct userfaultfd_ctx *ctx,
@@ -1322,15 +1330,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, vm_flags =0 ;if (uffdio_register.mode &UFFDIO_REGISTER_MODE_MISSING ) vm_flags |=VM_UFFD_MISSING ; -if (uffdio_register.mode &UFFDIO_REGISTER_MODE_WP ) { +if (uffdio_register.mode &UFFDIO_REGISTER_MODE_WP ) vm_flags |=VM_UFFD_WP ; -/* -* FIXME: remove the below error constraint by -* implementing the wprotect tracking mode. -*/ - ret = -EINVAL ; - goto out; - } ret =validate_range (mm, &uffdio_register.range.start, uffdio_register.range.len);
@@ -1380,7 +1381,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,/* check not compatible vmas */ ret = -EINVAL ; -if (!vma_can_userfault (cur)) +if (!vma_can_userfault (cur, vm_flags)) goto out_unlock;/*
@@ -1408,6 +1409,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,if (end & (vma_hpagesize -1 )) goto out_unlock; } +if ((vm_flags &VM_UFFD_WP ) && !(cur->vm_flags &VM_MAYWRITE )) + goto out_unlock;/* * Check that this vma isn't already owned by a
@@ -1437,7 +1440,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, do { cond_resched(); - BUG_ON(!vma_can_userfault(vma)); + BUG_ON(!vma_can_userfault(vma, vm_flags)); BUG_ON(vma->vm_userfaultfd_ctx.ctx && vma->vm_userfaultfd_ctx.ctx != ctx); WARN_ON(!(vma->vm_flags & VM_MAYWRITE));
@@ -1575,7 +1578,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,* provides for more strict behavior to notice * unregistration errors. */ -if (!vma_can_userfault (cur)) +if (!vma_can_userfault (cur, cur->vm_flags)) goto out_unlock; found = true;
@@ -1589,7 +1592,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, do { cond_resched(); - BUG_ON(!vma_can_userfault(vma)); + BUG_ON(!vma_can_userfault(vma, vma->vm_flags)); /* * Nothing to do: this vma is already registered into this
@@ -1802,6 +1805,50 @@ static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,return ret; } +static int userfaultfd_writeprotect (struct userfaultfd_ctx *ctx, +unsigned long arg) +{ +int ret; +struct uffdio_writeprotect uffdio_wp; +struct uffdio_writeprotect __user *user_uffdio_wp; +struct userfaultfd_wake_range range; + +if (READ_ONCE (ctx->mmap_changing)) +return -EAGAIN ; + + user_uffdio_wp = (struct uffdio_writeprotect __user *) arg; + +if (copy_from_user (&uffdio_wp, user_uffdio_wp, +sizeof (struct uffdio_writeprotect ))) +return -EFAULT ; + + ret =validate_range (ctx->mm, &uffdio_wp.range.start, + uffdio_wp.range.len); +if (ret) +return ret; + +if (uffdio_wp.mode & ~(UFFDIO_WRITEPROTECT_MODE_DONTWAKE | +UFFDIO_WRITEPROTECT_MODE_WP )) +return -EINVAL ; +if ((uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_WP ) && + (uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_DONTWAKE )) +return -EINVAL ; + + ret =mwriteprotect_range (ctx->mm, uffdio_wp.range.start, + uffdio_wp.range.len, uffdio_wp.mode & +UFFDIO_WRITEPROTECT_MODE_WP , + &ctx->mmap_changing); +if (ret) +return ret; + +if (!(uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_DONTWAKE )) { + range.start = uffdio_wp.range.start; + range.len = uffdio_wp.range.len; +wake_userfault (ctx, &range); + } +return ret; +} +static inline unsigned int uffd_ctx_features (__u64 user_features) {/*
@@ -1883,6 +1930,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,case UFFDIO_ZEROPAGE : ret =userfaultfd_zeropage (ctx, arg);break ; +case UFFDIO_WRITEPROTECT : + ret =userfaultfd_writeprotect (ctx, arg); +break ; }return ret; }
include/uapi/linux/userfaultfd.h +23 -0
@@ -52,6 +52,7 @@#define _UFFDIO_WAKE (0x02)#define _UFFDIO_COPY (0x03)#define _UFFDIO_ZEROPAGE (0x04) +#define _UFFDIO_WRITEPROTECT (0x06)#define _UFFDIO_API (0x3F)/* userfaultfd ioctl ids */
@@ -68,6 +69,8 @@ struct uffdio_copy)#define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ struct uffdio_zeropage) +#define UFFDIO_WRITEPROTECT _IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \ + struct uffdio_writeprotect)/* read() structure */ struct uffd_msg {
@@ -232,4 +235,24 @@ struct uffdio_zeropage { __s64 zeropage; }; +struct uffdio_writeprotect { +struct uffdio_range range; +/* +* UFFDIO_WRITEPROTECT_MODE_WP: set the flag to write protect a range, +* unset the flag to undo protection of a range which was previously +* write protected. +* +* UFFDIO_WRITEPROTECT_MODE_DONTWAKE: set the flag to avoid waking up +* any wait thread after the operation succeeds. +* +* NOTE: Write protecting a region (WP=1) is unrelated to page faults, +* therefore DONTWAKE flag is meaningless with WP=1. Removing write +* protection (WP=0) in response to a page fault wakes the faulting +* task unless DONTWAKE is set. +*/ +#define UFFDIO_WRITEPROTECT_MODE_WP ((__u64)1<<0) +#define UFFDIO_WRITEPROTECT_MODE_DONTWAKE ((__u64)1<<1) +__u64 mode; +}; +#endif /* _LINUX_USERFAULTFD_H */
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 14/19 userfaultfd: wp: enabled write protection in userfaultfd API
In reply to Peter XuFrom: Shaohua Li <[email protected]>
Now it's safe to enable write protection in userfaultfd API
--- include/uapi/linux/userfaultfd.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
include/uapi/linux/userfaultfd.h +4 -2
@@ -19,7 +19,8 @@* means the userland is reading). */ #define UFFD_API ((__u64)0xAA) -#define UFFD_API_FEATURES (UFFD_FEATURE_EVENT_FORK | \ +#define UFFD_API_FEATURES (UFFD_FEATURE_PAGEFAULT_FLAG_WP | \ + UFFD_FEATURE_EVENT_FORK | \ UFFD_FEATURE_EVENT_REMAP | \ UFFD_FEATURE_EVENT_REMOVE | \ UFFD_FEATURE_EVENT_UNMAP | \
@@ -34,7 +35,8 @@#define UFFD_API_RANGE_IOCTLS \ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY | \ - (__u64)1 << _UFFDIO_ZEROPAGE) + (__u64)1 << _UFFDIO_ZEROPAGE | \ + (__u64)1 << _UFFDIO_WRITEPROTECT)#define UFFD_API_RANGE_IOCTLS_BASIC \ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY)
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 15/19 userfaultfd: wp: don't wake up when doing write protect
In reply to Peter XuIt does not make sense to try to wake up any waiting thread when we're write-protecting a memory region. Only wake up when resolving a write protected page fault.
--- fs/userfaultfd.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
fs/userfaultfd.c +8 -5
@@ -1812,6 +1812,7 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,struct uffdio_writeprotect uffdio_wp;struct uffdio_writeprotect __user *user_uffdio_wp;struct userfaultfd_wake_range range; +bool mode_wp, mode_dontwake;if (READ_ONCE (ctx->mmap_changing))return -EAGAIN ;
@@ -1830,18 +1831,20 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,if (uffdio_wp.mode & ~(UFFDIO_WRITEPROTECT_MODE_DONTWAKE |UFFDIO_WRITEPROTECT_MODE_WP ))return -EINVAL ; -if ((uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_WP ) && - (uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_DONTWAKE )) + + mode_wp = uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_WP ; + mode_dontwake = uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_DONTWAKE ; + +if (mode_wp && mode_dontwake)return -EINVAL ; ret =mwriteprotect_range (ctx->mm, uffdio_wp.range.start, - uffdio_wp.range.len, uffdio_wp.mode & -UFFDIO_WRITEPROTECT_MODE_WP , + uffdio_wp.range.len, mode_wp, &ctx->mmap_changing);if (ret)return ret; -if (!(uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_DONTWAKE )) { +if (!mode_wp && !mode_dontwake) { range.start = uffdio_wp.range.start; range.len = uffdio_wp.range.len;wake_userfault (ctx, &range);
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 16/19 userfaultfd: wp: UFFDIO_REGISTER_MODE_WP documentation update
In reply to Peter XuFrom: Martin Cracauer <[email protected]>
Adds documentation about the write protection support.
--- Documentation/admin-guide/mm/userfaultfd.rst | 51 ++++++++++++++++++++ 1 file changed, 51 insertions(+)
Documentation/admin-guide/mm/userfaultfd.rst +51 -0
@@ -108,6 +108,57 @@ UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an half copied page since it'll keep userfaulting until the copy has finished. +Notes: + +- If you requested UFFDIO_REGISTER_MODE_MISSING when registering then + you must provide some kind of page in your thread after reading from + the uffd. You must provide either UFFDIO_COPY or UFFDIO_ZEROPAGE. + The normal behavior of the OS automatically providing a zero page on + an annonymous mmaping is not in place. + +- None of the page-delivering ioctls default to the range that you + registered with. You must fill in all fields for the appropriate + ioctl struct including the range. + +- You get the address of the access that triggered the missing page + event out of a struct uffd_msg that you read in the thread from the + uffd. You can supply as many pages as you want with UFFDIO_COPY or + UFFDIO_ZEROPAGE. Keep in mind that unless you used DONTWAKE then + the first of any of those IOCTLs wakes up the faulting thread. + +- Be sure to test for all errors including (pollfd[0].revents & + POLLERR). This can happen, e.g. when ranges supplied were + incorrect. + +Write Protect Notifications +--------------------------- + +This is equivalent to (but faster than) using mprotect and a SIGSEGV +signal handler. + +Firstly you need to register a range with UFFDIO_REGISTER_MODE_WP. +Instead of using mprotect(2) you use ioctl(uffd, UFFDIO_WRITEPROTECT, +struct *uffdio_writeprotect) while mode = UFFDIO_WRITEPROTECT_MODE_WP +in the struct passed in. The range does not default to and does not +have to be identical to the range you registered with. You can write +protect as many ranges as you like (inside the registered range). +Then, in the thread reading from uffd the struct will have +msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP set. Now you send +ioctl(uffd, UFFDIO_WRITEPROTECT, struct *uffdio_writeprotect) again +while pagefault.mode does not have UFFDIO_WRITEPROTECT_MODE_WP set. +This wakes up the thread which will continue to run with writes. This +allows you to do the bookkeeping about the write in the uffd reading +thread before the ioctl. + +If you registered with both UFFDIO_REGISTER_MODE_MISSING and +UFFDIO_REGISTER_MODE_WP then you need to think about the sequence in +which you supply a page and undo write protect. Note that there is a +difference between writes into a WP area and into a !WP area. The +former will have UFFD_PAGEFAULT_FLAG_WP set, the latter +UFFD_PAGEFAULT_FLAG_WRITE. The latter did not fail on protection but +you still need to supply a page when UFFDIO_REGISTER_MODE_MISSING was +used. + QEMU/KVM ========
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 17/19 userfaultfd: wp: declare _UFFDIO_WRITEPROTECT conditionally
In reply to Peter XuOnly declare _UFFDIO_WRITEPROTECT if the user specified UFFDIO_REGISTER_MODE_WP and if all the checks passed. Then when the user registers regions with shmem/hugetlbfs we won't expose the new ioctl to them. Even with complete anonymous memory range, we'll only expose the new WP ioctl bit if the register mode has MODE_WP.
--- fs/userfaultfd.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
fs/userfaultfd.c +13 -3
@@ -1495,14 +1495,24 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,up_write (&mm->mmap_sem);mmput (mm);if (!ret) { +__u64 ioctls_out; + + ioctls_out = basic_ioctls ?UFFD_API_RANGE_IOCTLS_BASIC : +UFFD_API_RANGE_IOCTLS ; + +/* +* Declare the WP ioctl only if the WP mode is +* specified and all checks passed with the range +*/ +if (!(uffdio_register.mode &UFFDIO_REGISTER_MODE_WP )) + ioctls_out &= ~((__u64 )1 << _UFFDIO_WRITEPROTECT); +/* * Now that we scanned all vmas we can already tell * userland which ioctls methods are guaranteed to * succeed on this range. */ -if (put_user (basic_ioctls ?UFFD_API_RANGE_IOCTLS_BASIC : -UFFD_API_RANGE_IOCTLS , - &user_uffdio_register->ioctls)) +if (put_user (ioctls_out, &user_uffdio_register->ioctls)) ret = -EFAULT ; } out:
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 18/19 userfaultfd: selftests: refactor statistics
In reply to Peter XuIntroduce uffd_stats structure for statistics of the self test, at the same time refactor the code to always pass in the uffd_stats for either read() or poll() typed fault handling threads instead of using two different ways to return the statistic results. No functional change.
With the new structure, it's very easy to introduce new statistics.
--- tools/testing/selftests/vm/userfaultfd.c | 76 +++++++++++++++--------- 1 file changed, 49 insertions(+), 27 deletions(-)
tools/testing/selftests/vm/userfaultfd.c +49 -27
@@ -86,6 +86,12 @@ static char *area_src, *area_src_alias, *area_dst, *area_dst_alias;static char *zeropage;pthread_attr_t attr; +/* Userfaultfd test statistics */ +struct uffd_stats { +int cpu; +unsigned long missing_faults; +}; +/* pthread_mutex_t starts at page offset 0 */ #define area_mutex (___area, ___nr) \ ((pthread_mutex_t *) ((___area) + (___nr)*page_size))
@@ -125,6 +131,17 @@ static void usage(void)exit (1 ); } +static void uffd_stats_reset (struct uffd_stats *uffd_stats, +unsigned long n_cpus) +{ +int i; + +for (i =0 ; i < n_cpus; i++) { + uffd_stats[i].cpu = i; + uffd_stats[i].missing_faults =0 ; + } +} +static int anon_release_pages (char *rel_area) {int ret =0 ;
@@ -467,8 +484,8 @@ static int uffd_read_msg(int ufd, struct uffd_msg *msg)return 0 ; } -/* Return 1 if page fault handled by us; otherwise 0 */ -static int uffd_handle_page_fault (struct uffd_msg *msg) +static void uffd_handle_page_fault (struct uffd_msg *msg, +struct uffd_stats *stats) {unsigned long offset;
@@ -483,18 +500,19 @@ static int uffd_handle_page_fault(struct uffd_msg *msg) offset = (char *)(unsigned long )msg->arg.pagefault.address - area_dst; offset &= ~(page_size-1 ); -return copy_page (uffd, offset); +if (copy_page (uffd, offset)) + stats->missing_faults++; }static void *uffd_poll_thread (void *arg) { -unsigned long cpu = (unsigned long ) arg; +struct uffd_stats *stats = (struct uffd_stats *)arg; +unsigned long cpu = stats->cpu;struct pollfd pollfd[2 ];struct uffd_msg msg;struct uffdio_register uffd_reg;int ret;char tmp_chr; -unsigned long userfaults =0 ; pollfd[0 ].fd = uffd; pollfd[0 ].events =POLLIN ;
@@ -524,7 +542,7 @@ static void *uffd_poll_thread(void *arg) msg.event), exit(1);break ;case UFFD_EVENT_PAGEFAULT : - userfaults +=uffd_handle_page_fault (&msg); +uffd_handle_page_fault (&msg, stats);break ;case UFFD_EVENT_FORK :close (uffd);
@@ -543,28 +561,27 @@ static void *uffd_poll_thread(void *arg)break ; } } -return (void *)userfaults; + +return NULL ; }pthread_mutex_t uffd_read_mutex =PTHREAD_MUTEX_INITIALIZER ;static void *uffd_read_thread (void *arg) { -unsigned long *this_cpu_userfaults; +struct uffd_stats *stats = (struct uffd_stats *)arg;struct uffd_msg msg; - this_cpu_userfaults = (unsigned long *) arg; - *this_cpu_userfaults =0 ; -pthread_mutex_unlock (&uffd_read_mutex);/* from here cancellation is ok */ for (;;) {if (uffd_read_msg (uffd, &msg))continue ; - (*this_cpu_userfaults) +=uffd_handle_page_fault (&msg); +uffd_handle_page_fault (&msg, stats); } -return (void *)NULL ; + +return NULL ; }static void *background_thread (void *arg)
@@ -580,13 +597,12 @@ static void *background_thread(void *arg)return NULL ; } -static int stress (unsigned long *userfaults) +static int stress (struct uffd_stats *uffd_stats) {unsigned long cpu;pthread_t locking_threads[nr_cpus];pthread_t uffd_threads[nr_cpus];pthread_t background_threads[nr_cpus]; -void **_userfaults = (void **) userfaults; finished =0 ;for (cpu =0 ; cpu < nr_cpus; cpu++) {
@@ -595,12 +611,13 @@ static int stress(unsigned long *userfaults)return 1 ;if (bounces &BOUNCE_POLL ) {if (pthread_create (&uffd_threads[cpu], &attr, - uffd_poll_thread, (void *)cpu)) + uffd_poll_thread, + (void *)&uffd_stats[cpu]))return 1 ; }else {if (pthread_create (&uffd_threads[cpu], &attr, uffd_read_thread, - &_userfaults[cpu])) + (void *)&uffd_stats[cpu]))return 1 ;pthread_mutex_lock (&uffd_read_mutex); }
@@ -637,7 +654,8 @@ static int stress(unsigned long *userfaults)fprintf (stderr,"pipefd write error\n" );return 1 ; } -if (pthread_join (uffd_threads[cpu], &_userfaults[cpu])) +if (pthread_join (uffd_threads[cpu], + (void *)&uffd_stats[cpu]))return 1 ; }else {if (pthread_cancel (uffd_threads[cpu]))
@@ -908,11 +926,11 @@ static int userfaultfd_events_test(void) {struct uffdio_register uffdio_register;unsigned long expected_ioctls; -unsigned long userfaults;pthread_t uffd_mon;int err, features;pid_t pid;char c; +struct uffd_stats stats = {0 };printf ("testing events (fork, remap, remove): " );fflush (stdout);
@@ -939,7 +957,7 @@ static int userfaultfd_events_test(void) "unexpected missing ioctl for anon memory\n"),exit (1 ); -if (pthread_create (&uffd_mon, &attr, uffd_poll_thread,NULL )) +if (pthread_create (&uffd_mon, &attr, uffd_poll_thread, &stats))perror ("uffd_poll_thread create" ),exit (1 ); pid =fork ();
@@ -955,13 +973,13 @@ static int userfaultfd_events_test(void)if (write (pipefd[1 ], &c,sizeof (c)) !=sizeof (c))perror ("pipe write" ),exit (1 ); -if (pthread_join (uffd_mon, (void **)&userfaults)) +if (pthread_join (uffd_mon,NULL ))return 1 ;close (uffd); -printf ("userfaults: %ld\n" , userfaults); +printf ("userfaults: %ld\n" , stats.missing_faults); -return userfaults != nr_pages; +return stats.missing_faults != nr_pages; }static int userfaultfd_sig_test (void )
@@ -973,6 +991,7 @@ static int userfaultfd_sig_test(void)int err, features;pid_t pid;char c; +struct uffd_stats stats = {0 };printf ("testing signal delivery: " );fflush (stdout);
@@ -1004,7 +1023,7 @@ static int userfaultfd_sig_test(void)if (uffd_test_ops->release_pages (area_dst))return 1 ; -if (pthread_create (&uffd_mon, &attr, uffd_poll_thread,NULL )) +if (pthread_create (&uffd_mon, &attr, uffd_poll_thread, &stats))perror ("uffd_poll_thread create" ),exit (1 ); pid =fork ();
@@ -1030,6 +1049,7 @@ static int userfaultfd_sig_test(void)close (uffd);return userfaults !=0 ; } +static int userfaultfd_stress (void ) {void *area;
@@ -1038,7 +1058,7 @@ static int userfaultfd_stress(void)struct uffdio_register uffdio_register;unsigned long cpu;int err; -unsigned long userfaults[nr_cpus]; +struct uffd_stats uffd_stats[nr_cpus]; uffd_test_ops->allocate_area ((void **)&area_src);if (!area_src)
@@ -1167,8 +1187,10 @@ static int userfaultfd_stress(void)if (uffd_test_ops->release_pages (area_dst))return 1 ; +uffd_stats_reset (uffd_stats, nr_cpus); +/* bounce pass */ -if (stress (userfaults)) +if (stress (uffd_stats))return 1 ;/* unregister */
@@ -1211,7 +1233,7 @@ static int userfaultfd_stress(void)printf ("userfaults:" );for (cpu =0 ; cpu < nr_cpus; cpu++) -printf (" %lu" , userfaults[cpu]); +printf (" %lu" , uffd_stats[cpu].missing_faults);printf ("\n" ); }
Signature · 1 line
2.24.1
Peter Xu
reviewerPATCH v6 19/19 userfaultfd: selftests: add write-protect test
In reply to Peter XuThis patch adds uffd tests for write protection.
Instead of introducing new tests for it, let's simply squashing uffd-wp tests into existing uffd-missing test cases. Changes are:
(1) Bouncing tests
We do the write-protection in two ways during the bouncing test:
- By using UFFDIO_COPY_MODE_WP when resolving MISSING pages: then we'll make sure for each bounce process every single page will be at least fault twice: once for MISSING, once for WP.
- By direct call UFFDIO_WRITEPROTECT on existing faulted memories:
To further torture the explicit page protection procedures of uffd-wp, we split each bounce procedure into two halves (in the background thread): the first half will be MISSING+WP for each page as explained above. After the first half, we write protect the faulted region in the background thread to make sure at least half of the pages will be write protected again which is the first half to test the new UFFDIO_WRITEPROTECT call. Then we continue with the 2nd half, which will contain both MISSING and WP faulting tests for the 2nd half and WP-only faults from the 1st half.
(2) Event/Signal test
Mostly previous tests but will do MISSING+WP for each page. For sigbus-mode test we'll need to provide standalone path to handle the write protection faults.
For all tests, do statistics as well for uffd-wp pages.
--- tools/testing/selftests/vm/userfaultfd.c | 157 +++++++++++++++++++---- 1 file changed, 133 insertions(+), 24 deletions(-)
tools/testing/selftests/vm/userfaultfd.c +133 -24
@@ -54,6 +54,7 @@#include <linux/userfaultfd.h> #include <setjmp.h> #include <stdbool.h> +#include <assert.h> #include "../kselftest.h"
@@ -76,6 +77,8 @@ static int test_type;#define ALARM_INTERVAL_SECS 10static volatile bool test_uffdio_copy_eexist = true;static volatile bool test_uffdio_zeropage_eexist = true; +/* Whether to test uffd write-protection */ +static bool test_uffdio_wp = false;static bool map_shared;static int huge_fd;
@@ -90,6 +93,7 @@ pthread_attr_t attr;struct uffd_stats {int cpu;unsigned long missing_faults; +unsigned long wp_faults; };/* pthread_mutex_t starts at page offset 0 */
@@ -139,9 +143,29 @@ static void uffd_stats_reset(struct uffd_stats *uffd_stats,for (i =0 ; i < n_cpus; i++) { uffd_stats[i].cpu = i; uffd_stats[i].missing_faults =0 ; + uffd_stats[i].wp_faults =0 ; } } +static void uffd_stats_report (struct uffd_stats *stats,int n_cpus) +{ +int i; +unsigned long long miss_total =0 , wp_total =0 ; + +for (i =0 ; i < n_cpus; i++) { + miss_total += stats[i].missing_faults; + wp_total += stats[i].wp_faults; + } + +printf ("userfaults: %llu missing (" , miss_total); +for (i =0 ; i < n_cpus; i++) +printf ("%lu+" , stats[i].missing_faults); +printf ("\b), %llu wp (" , wp_total); +for (i =0 ; i < n_cpus; i++) +printf ("%lu+" , stats[i].wp_faults); +printf ("\b)\n" ); +} +static int anon_release_pages (char *rel_area) {int ret =0 ;
@@ -262,10 +286,15 @@ struct uffd_test_ops {void (*alias_mapping)(__u64 *start,size_t len,unsigned long offset); }; -#define ANON_EXPECTED_IOCTLS ((1 << _UFFDIO_WAKE) | \ +#define SHMEM_EXPECTED_IOCTLS ((1 << _UFFDIO_WAKE) | \ (1 << _UFFDIO_COPY) | \ (1 << _UFFDIO_ZEROPAGE)) +#define ANON_EXPECTED_IOCTLS ((1 << _UFFDIO_WAKE) | \ + (1 << _UFFDIO_COPY) | \ + (1 << _UFFDIO_ZEROPAGE) | \ + (1 << _UFFDIO_WRITEPROTECT)) +static struct uffd_test_ops anon_uffd_test_ops = { .expected_ioctls =ANON_EXPECTED_IOCTLS , .allocate_area = anon_allocate_area,
@@ -274,7 +303,7 @@ static struct uffd_test_ops anon_uffd_test_ops = { };static struct uffd_test_ops shmem_uffd_test_ops = { - .expected_ioctls =ANON_EXPECTED_IOCTLS , + .expected_ioctls =SHMEM_EXPECTED_IOCTLS , .allocate_area = shmem_allocate_area, .release_pages = shmem_release_pages, .alias_mapping = noop_alias_mapping,
@@ -298,6 +327,21 @@ static int my_bcmp(char *str1, char *str2, size_t n)return 0 ; } +static void wp_range (int ufd,__u64 start,__u64 len,bool wp) +{ +struct uffdio_writeprotect prms = {0 }; + +/* Write protection page faults */ + prms.range.start = start; + prms.range.len = len; +/* Undo write-protect, do wakeup after that */ + prms.mode = wp ?UFFDIO_WRITEPROTECT_MODE_WP :0 ; + +if (ioctl (ufd,UFFDIO_WRITEPROTECT , &prms)) +fprintf (stderr,"clear WP failed for address 0x%Lx\n" , + start),exit (1 ); +} +static void *locking_thread (void *arg) {unsigned long cpu = (unsigned long ) arg;
@@ -436,7 +480,10 @@ static int __copy_page(int ufd, unsigned long offset, bool retry) uffdio_copy.dst = (unsigned long ) area_dst + offset; uffdio_copy.src = (unsigned long ) area_src + offset; uffdio_copy.len = page_size; - uffdio_copy.mode =0 ; +if (test_uffdio_wp) + uffdio_copy.mode =UFFDIO_COPY_MODE_WP ; +else + uffdio_copy.mode =0 ; uffdio_copy.copy =0 ;if (ioctl (ufd,UFFDIO_COPY , &uffdio_copy)) {/* real retval in ufdio_copy.copy */
@@ -493,15 +540,21 @@ static void uffd_handle_page_fault(struct uffd_msg *msg,fprintf (stderr,"unexpected msg event %u\n" , msg->event),exit (1 ); -if (bounces &BOUNCE_VERIFY && - msg->arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WRITE ) -fprintf (stderr,"unexpected write fault\n" ),exit (1 ); +if (msg->arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WP ) { +wp_range (uffd, msg->arg.pagefault.address, page_size, false); + stats->wp_faults++; + }else { +/* Missing page faults */ +if (bounces &BOUNCE_VERIFY && + msg->arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WRITE ) +fprintf (stderr,"unexpected write fault\n" ),exit (1 ); - offset = (char *)(unsigned long )msg->arg.pagefault.address - area_dst; - offset &= ~(page_size-1 ); + offset = (char *)(unsigned long )msg->arg.pagefault.address - area_dst; + offset &= ~(page_size-1 ); -if (copy_page (uffd, offset)) - stats->missing_faults++; +if (copy_page (uffd, offset)) + stats->missing_faults++; + } }static void *uffd_poll_thread (void *arg)
@@ -587,11 +640,30 @@ static void *uffd_read_thread(void *arg)static void *background_thread (void *arg) {unsigned long cpu = (unsigned long ) arg; -unsigned long page_nr; +unsigned long page_nr, start_nr, mid_nr, end_nr; + + start_nr = cpu * nr_pages_per_cpu; + end_nr = (cpu+1 ) * nr_pages_per_cpu; + mid_nr = (start_nr + end_nr) /2 ; + +/* Copy the first half of the pages */ +for (page_nr = start_nr; page_nr < mid_nr; page_nr++) +copy_page_retry (uffd, page_nr * page_size); -for (page_nr = cpu * nr_pages_per_cpu; - page_nr < (cpu+1 ) * nr_pages_per_cpu; - page_nr++) +/* +* If we need to test uffd-wp, set it up now. Then we'll have +* at least the first half of the pages mapped already which +* can be write-protected for testing +*/ +if (test_uffdio_wp) +wp_range (uffd, (unsigned long )area_dst + start_nr * page_size, + nr_pages_per_cpu * page_size, true); + +/* +* Continue the 2nd half of the page copying, handling write +* protection faults if any +*/ +for (page_nr = mid_nr; page_nr < end_nr; page_nr++)copy_page_retry (uffd, page_nr * page_size);return NULL ;
@@ -753,17 +825,31 @@ static int faulting_process(int signal_test) }for (nr =0 ; nr < split_nr_pages; nr++) { +int steps =1 ; +unsigned long offset = nr * page_size; +if (signal_test) {if (sigsetjmp (*sigbuf,1 ) !=0 ) { -if (nr == lastnr) { +if (steps ==1 && nr == lastnr) {fprintf (stderr,"Signal repeated\n" );return 1 ; } lastnr = nr;if (signal_test ==1 ) { -if (copy_page (uffd, nr * page_size)) - signalled++; +if (steps ==1 ) { +/* This is a MISSING request */ + steps++; +if (copy_page (uffd, offset)) + signalled++; + }else { +/* This is a WP request */ +assert (steps ==2 ); +wp_range (uffd, + (__u64 )area_dst + + offset, + page_size, false); + } }else { signalled++;continue ;
@@ -776,8 +862,13 @@ static int faulting_process(int signal_test)fprintf (stderr,"nr %lu memory corruption %Lu %Lu\n" , nr, count, - count_verify[nr]),exit (1 ); - } + count_verify[nr]); + } +/* +* Trigger write protection if there is by writting +* the same value back. +*/ + *area_count (area_dst, nr) = count; }if (signal_test)
@@ -799,6 +890,11 @@ static int faulting_process(int signal_test) nr, count, count_verify[nr]), exit(1); } +/* +* Trigger write protection if there is by writting +* the same value back. +*/ + *area_count (area_dst, nr) = count; }if (uffd_test_ops->release_pages (area_dst))
@@ -902,6 +998,8 @@ static int userfaultfd_zeropage_test(void) uffdio_register.range.start = (unsigned long ) area_dst; uffdio_register.range.len = nr_pages * page_size; uffdio_register.mode =UFFDIO_REGISTER_MODE_MISSING ; +if (test_uffdio_wp) + uffdio_register.mode |=UFFDIO_REGISTER_MODE_WP ;if (ioctl (uffd,UFFDIO_REGISTER , &uffdio_register))fprintf (stderr,"register failure\n" ),exit (1 );
@@ -947,6 +1045,8 @@ static int userfaultfd_events_test(void) uffdio_register.range.start = (unsigned long ) area_dst; uffdio_register.range.len = nr_pages * page_size; uffdio_register.mode =UFFDIO_REGISTER_MODE_MISSING ; +if (test_uffdio_wp) + uffdio_register.mode |=UFFDIO_REGISTER_MODE_WP ;if (ioctl (uffd,UFFDIO_REGISTER , &uffdio_register))fprintf (stderr,"register failure\n" ),exit (1 );
@@ -977,7 +1077,8 @@ static int userfaultfd_events_test(void)return 1 ;close (uffd); -printf ("userfaults: %ld\n" , stats.missing_faults); + +uffd_stats_report (&stats,1 );return stats.missing_faults != nr_pages; }
@@ -1007,6 +1108,8 @@ static int userfaultfd_sig_test(void) uffdio_register.range.start = (unsigned long ) area_dst; uffdio_register.range.len = nr_pages * page_size; uffdio_register.mode =UFFDIO_REGISTER_MODE_MISSING ; +if (test_uffdio_wp) + uffdio_register.mode |=UFFDIO_REGISTER_MODE_WP ;if (ioctl (uffd,UFFDIO_REGISTER , &uffdio_register))fprintf (stderr,"register failure\n" ),exit (1 );
@@ -1139,6 +1242,8 @@ static int userfaultfd_stress(void) uffdio_register.range.start = (unsigned long ) area_dst; uffdio_register.range.len = nr_pages * page_size; uffdio_register.mode =UFFDIO_REGISTER_MODE_MISSING ; +if (test_uffdio_wp) + uffdio_register.mode |=UFFDIO_REGISTER_MODE_WP ;if (ioctl (uffd,UFFDIO_REGISTER , &uffdio_register)) {fprintf (stderr,"register failure\n" );return 1 ;
@@ -1193,6 +1298,11 @@ static int userfaultfd_stress(void)if (stress (uffd_stats))return 1 ; +/* Clear all the write protections if there is any */ +if (test_uffdio_wp) +wp_range (uffd, (unsigned long )area_dst, + nr_pages * page_size, false); +/* unregister */ if (ioctl (uffd,UFFDIO_UNREGISTER , &uffdio_register.range)) {fprintf (stderr,"unregister failure\n" );
@@ -1231,10 +1341,7 @@ static int userfaultfd_stress(void) area_src_alias = area_dst_alias; area_dst_alias = tmp_area; -printf ("userfaults:" ); -for (cpu =0 ; cpu < nr_cpus; cpu++) -printf (" %lu" , uffd_stats[cpu].missing_faults); -printf ("\n" ); +uffd_stats_report (uffd_stats, nr_cpus); }if (err)
@@ -1274,6 +1381,8 @@ static void set_test_type(const char *type)if (!strcmp (type,"anon" )) { test_type =TEST_ANON ; uffd_test_ops = &anon_uffd_test_ops; +/* Only enable write-protect test for anonymous test */ + test_uffdio_wp = true; }else if (!strcmp (type,"hugetlb" )) { test_type =TEST_HUGETLB ; uffd_test_ops = &hugetlb_uffd_test_ops;
Signature · 1 line
2.24.1