userfaultfd: add write protect support
Subsystems (24)
cover userfaultfd: add write protect support
Hi,
There is plan to support write protect fault into userfaultfd before, but it's not implemented yet. I'm working on a library to support different types of buffer like compressed buffer and file buffer, something like a page cache implementation in userspace. The buffer enables userfaultfd and does something like decompression in userfault handler. When memory size exceeds a threshold, madvise is used to reclaim memory. The problem is data can be corrupted in reclaim without memory protection support.
For example, in the compressed buffer case, reclaim does:
1. compress memory range and store compressed data elsewhere
2. madvise the memory range
But if the memory is changed before 2, new change is lost. memory write protection can solve the issue. With it, the reclaim does:
1. write protect memory range
2. compress memory range and store compressed data elsewhere
3. madvise the memory range
4. undo write protect memory range and wakeup tasks waiting in write protect fault.
If a task changes memory before 3, write protect fault will be triggered. we can put the task into sleep till step 4 runs for example. In this way memory changes will not be lost.
This patch set add write protect support for userfaultfd. One issue is write protect fault can happen even without enabling write protect in userfault. For example, a write to address backed by zero page. There is no way to distinguish if this is a write protect fault expected by userfault. This patch just blindly triggers write protect fault to userfault if corresponding vma enables VM_UFFD_WP. Application should be prepared to handle such write protect fault.
Thanks,
Shaohua
Shaohua Li (8):
userfaultfd: add helper for writeprotect check
userfaultfd: support write protection for userfault vma range
userfaultfd: expose writeprotect API to ioctl
userfaultfd: allow userfaultfd register success with writeprotection
userfaultfd: undo write proctection in unregister
userfaultfd: hook userfault handler to write protection fault
userfaultfd: fault try one more time
userfaultfd: enabled write protection in userfaultfd API
arch/alpha/mm/fault.c | 8 ++++- arch/arc/mm/fault.c | 8 ++++- arch/arm/mm/fault.c | 8 ++++- arch/arm64/mm/fault.c | 8 ++++- arch/avr32/mm/fault.c | 8 ++++- arch/cris/mm/fault.c | 8 ++++- arch/hexagon/mm/vm_fault.c | 8 ++++- arch/ia64/mm/fault.c | 8 ++++- arch/m68k/mm/fault.c | 8 ++++- arch/metag/mm/fault.c | 8 ++++- arch/microblaze/mm/fault.c | 8 ++++- arch/mips/mm/fault.c | 8 ++++- arch/mn10300/mm/fault.c | 8 ++++- arch/nios2/mm/fault.c | 8 ++++- arch/openrisc/mm/fault.c | 8 ++++- arch/parisc/mm/fault.c | 8 ++++- arch/powerpc/mm/fault.c | 8 ++++- arch/s390/mm/fault.c | 9 +++++- arch/sh/mm/fault.c | 8 ++++- arch/sparc/mm/fault_32.c | 8 ++++- arch/sparc/mm/fault_64.c | 8 ++++- arch/tile/mm/fault.c | 8 ++++- arch/um/kernel/trap.c | 8 ++++- arch/unicore32/mm/fault.c | 8 ++++- arch/x86/mm/fault.c | 9 +++++- arch/xtensa/mm/fault.c | 8 ++++- fs/userfaultfd.c | 66 ++++++++++++++++++++++++++++++++++------ include/linux/mm.h | 3 +- include/linux/userfaultfd_k.h | 12 ++++++++ include/uapi/linux/userfaultfd.h | 17 +++++++++-- mm/memory.c | 66 +++++++++++++++++++++++++++++----------- mm/userfaultfd.c | 52 +++++++++++++++++++++++++++++++ 32 files changed, 369 insertions(+), 57 deletions(-)
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 1/8 userfaultfd: add helper for writeprotect check
add helper for writeprotect check. Will use it later.
--- include/linux/userfaultfd_k.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
include/linux/userfaultfd_k.h +10 -0
@@ -48,6 +48,11 @@ static inline bool userfaultfd_missing(struct vm_area_struct *vma)return vma->vm_flags &VM_UFFD_MISSING ; } +static inline bool userfaultfd_wp (struct vm_area_struct *vma) +{ +return vma->vm_flags &VM_UFFD_WP ; +} +static inline bool userfaultfd_armed (struct vm_area_struct *vma) {return vma->vm_flags & (VM_UFFD_MISSING |VM_UFFD_WP );
@@ -75,6 +80,11 @@ static inline bool userfaultfd_missing(struct vm_area_struct *vma)return false; } +static inline bool userfaultfd_wp (struct vm_area_struct *vma) +{ +return false; +} +static inline bool userfaultfd_armed (struct vm_area_struct *vma) {return false;
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 2/8 userfaultfd: support write protection for userfault vma range
In reply to Shaohua LiAdd API to enable/disable writeprotect a vma range. Unlike mprotect, this doesn't split/merge vmas.
--- include/linux/userfaultfd_k.h | 2 ++ mm/userfaultfd.c | 52 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+)
include/linux/userfaultfd_k.h +2 -0
@@ -35,6 +35,8 @@ extern ssize_t mcopy_atomic(struct mm_struct *dst_mm, unsigned long dst_start,extern ssize_t mfill_zeropage (struct mm_struct *dst_mm,unsigned long dst_start,unsigned long len); +extern int mwriteprotect_range (struct mm_struct *dst_mm, +unsigned long start,unsigned long len,bool enable_wp);/* mm helpers */ static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,
mm/userfaultfd.c +52 -0
55 lines not on this page · raw message
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 2/8] userfaultfd: support write protection for userfault vma range
Hello,
Do you have a more recent version of this patchset?
Quoted 1 line · Shaohua Li
mm/userfaultfd.c +1
-0+down_read (&dst_mm->mmap_sem);
[..]
mm/userfaultfd.c ssize_t mfill_zeropage(struct mm_struct *dst_mm, unsigned long start, · line 351
Quoted 4 lines
mm/userfaultfd.c +4
-0+if (enable_wp) + newprot =vm_get_page_prot (dst_vma->vm_flags & ~(VM_WRITE )); +else + newprot =vm_get_page_prot (dst_vma->vm_flags);
The vm_flags for anon vmas are always wrprotected, just we mark them writable during fault or during cow if vm_flags VM_WRITE is set, when we know it's not shared. So this requires checking the mapcount somewhere while fork cannot run, or the above won't properly unprotect?
mm/userfaultfd.c ssize_t mfill_zeropage(struct mm_struct *dst_mm, unsigned long start, · line 354
Quoted 3 lines
mm/userfaultfd.c +3
-0+ +change_protection (dst_vma, start, start + len, newprot, + !enable_wp,0 );
change_protection(prot_numa=0) assumes mmap_sem hold for writing breaking here:
/* !prot_numa is protected by mmap_sem held for write */
if (!prot_numa)
return pte_offset_map_lock(vma->vm_mm, pmd, addr, ptl);
pmdl = pmd_lock(vma->vm_mm, pmd);
if (unlikely(pmd_trans_huge(*pmd) || pmd_none(*pmd))) {
spin_unlock(pmdl);
return NULL;
}
pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, ptl);
spin_unlock(pmdl);With userfaultfd the pmd can be trans unstable as we only hold the mmap_sem for reading.
In short calling change_protection() with prot_numa==0 with only the mmap_sem for reading looks wrong...
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 3/8 userfaultfd: expose writeprotect API to ioctl
In reply to Shaohua LiAdd the writeprotect API to userfaultfd ioctl
--- fs/userfaultfd.c | 45 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/userfaultfd.h | 10 +++++++++ 2 files changed, 55 insertions(+)
fs/userfaultfd.c +45 -0
@@ -1122,6 +1122,49 @@ static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,return ret; } +static int userfaultfd_writeprotect (struct userfaultfd_ctx *ctx, +unsigned long arg) +{ +int ret; +struct uffdio_writeprotect uffdio_wp; +struct uffdio_writeprotect __user *user_uffdio_wp; +struct userfaultfd_wake_range range; + + user_uffdio_wp = (struct uffdio_writeprotect __user *) arg; + +if (copy_from_user (&uffdio_wp, user_uffdio_wp, +sizeof (struct uffdio_writeprotect ))) +return -EFAULT ; + + ret =validate_range (ctx->mm, uffdio_wp.range.start, + uffdio_wp.range.len); +if (ret) +return ret; + +if (uffdio_wp.mode & ~(UFFDIO_WRITEPROTECT_MODE_DONTWAKE | +UFFDIO_WRITEPROTECT_MODE_WP )) +return -EINVAL ; +if ((uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_WP ) && + (uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_DONTWAKE )) +return -EINVAL ; + +if (uffdio_wp.mode &UFFDIO_WRITEPROTECT_MODE_WP ) + ret =mwriteprotect_range (ctx->mm, uffdio_wp.range.start, + uffdio_wp.range.len, true); +else + ret =mwriteprotect_range (ctx->mm, uffdio_wp.range.start, + uffdio_wp.range.len, false); +if (ret) +return ret; + +if (!(uffdio_wp.mode &UFFDIO_COPY_MODE_DONTWAKE )) { + range.start = uffdio_wp.range.start; + range.len = uffdio_wp.range.len; +wake_userfault (ctx, &range); + } +return ret; +} +/* * userland asks for a certain API version and we return which bits * and ioctl commands are implemented in this kernel for such API
@@ -1186,6 +1229,8 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,case UFFDIO_ZEROPAGE : ret =userfaultfd_zeropage (ctx, arg);break ; +case UFFDIO_WRITEPROTECT : + ret =userfaultfd_writeprotect (ctx, arg); }return ret; }
include/uapi/linux/userfaultfd.h +10 -0
@@ -40,6 +40,7 @@#define _UFFDIO_WAKE (0x02)#define _UFFDIO_COPY (0x03)#define _UFFDIO_ZEROPAGE (0x04) +#define _UFFDIO_WRITEPROTECT (0x05)#define _UFFDIO_API (0x3F)/* userfaultfd ioctl ids */
@@ -56,6 +57,8 @@ struct uffdio_copy)#define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ struct uffdio_zeropage) +#define UFFDIO_WRITEPROTECT _IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \ + struct uffdio_writeprotect)/* read() structure */ struct uffd_msg {
@@ -164,4 +167,11 @@ struct uffdio_zeropage { __s64 zeropage; }; +struct uffdio_writeprotect { +struct uffdio_range range; +/* !WP means undo writeprotect. DONTWAKE is valid only with !WP */ +#define UFFDIO_WRITEPROTECT_MODE_WP ((__u64)1<<0) +#define UFFDIO_WRITEPROTECT_MODE_DONTWAKE ((__u64)1<<1) +__u64 mode; +};#endif /* _LINUX_USERFAULTFD_H */
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 4/8 userfaultfd: allow userfaultfd register success with writeprotection
In reply to Shaohua LiThe userfaultfd register ioctl currently disables writeprotection. Enable it.
--- fs/userfaultfd.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-)
fs/userfaultfd.c +5 -8
@@ -236,6 +236,8 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,*/ if (pte_none (*pte)) ret = true; +if (!pte_write (*pte) && (reason &VM_UFFD_WP )) + ret = true;pte_unmap (pte); out:
@@ -736,15 +738,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, vm_flags =0 ;if (uffdio_register.mode &UFFDIO_REGISTER_MODE_MISSING ) vm_flags |=VM_UFFD_MISSING ; -if (uffdio_register.mode &UFFDIO_REGISTER_MODE_WP ) { +if (uffdio_register.mode &UFFDIO_REGISTER_MODE_WP ) vm_flags |=VM_UFFD_WP ; -/* -* FIXME: remove the below error constraint by -* implementing the wprotect tracking mode. -*/ - ret = -EINVAL ; - goto out; - } ret =validate_range (mm, uffdio_register.range.start, uffdio_register.range.len);
@@ -784,6 +779,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, ret = -EINVAL ;if (cur->vm_ops) goto out_unlock; +if ((vm_flags &VM_UFFD_WP ) && !(cur->vm_flags &VM_WRITE )) + goto out_unlock;/* * Check that this vma isn't already owned by a
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 5/8 userfaultfd: undo write proctection in unregister
In reply to Shaohua LiAfter a userfaultfd unregister, make sure the range doesn't disable write in ptes.
--- fs/userfaultfd.c | 3 +++ 1 file changed, 3 insertions(+)
fs/userfaultfd.c +3 -0
@@ -953,6 +953,9 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,if (vma->vm_start > start) start = vma->vm_start; vma_end =min (end, vma->vm_end); +if (userfaultfd_wp (vma)) +change_protection (vma, start, vma_end, +vm_get_page_prot (vma->vm_flags),1 ,0 ); new_flags = vma->vm_flags & ~(VM_UFFD_MISSING |VM_UFFD_WP ); prev = vma_merge(mm, prev, start, vma_end, new_flags,
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 6/8 userfaultfd: hook userfault handler to write protection fault
In reply to Shaohua LiThere are several cases write protection fault happens. It could be a write to zero page, swaped page or userfault write protected page. When the fault happens, there is no way to know if userfault write protect the page before. Here we just blindly issue a userfault notification for vma with VM_UFFD_WP regardless if app write protects it yet. Application should be ready to handle such wp fault.
--- mm/memory.c | 66 +++++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 17 deletions(-)
mm/memory.c +49 -17
@@ -1994,10 +1994,11 @@ static inline int wp_page_reuse(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, pte_t *page_table, spinlock_t *ptl, pte_t orig_pte, struct page *page, int page_mkwrite, - int dirty_shared) + int dirty_shared, unsigned int flags) __releases(ptl) { pte_t entry; +bool do_uffd = false;/* * Clear the pages cpupid information as the existing * information potentially belongs to a now completely
@@ -2008,10 +2009,16 @@ static inline int wp_page_reuse(struct mm_struct *mm,flush_cache_page (vma, address,pte_pfn (orig_pte)); entry =pte_mkyoung (orig_pte); - entry =maybe_mkwrite (pte_mkdirty (entry), vma); +if (userfaultfd_wp (vma) && page) { + entry =pte_mkdirty (entry); + do_uffd = true; + }else + entry =maybe_mkwrite (pte_mkdirty (entry), vma);if (ptep_set_access_flags (vma, address, page_table, entry,1 ))update_mmu_cache (vma, address, page_table);pte_unmap_unlock (page_table, ptl); +if (do_uffd) +return handle_userfault (vma, address, flags,VM_UFFD_WP );if (dirty_shared) {struct address_space *mapping;
@@ -2059,7 +2066,7 @@ static inline int wp_page_reuse(struct mm_struct *mm,*/ static int wp_page_copy (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *page_table,pmd_t *pmd, -pte_t orig_pte,struct page *old_page) +pte_t orig_pte,struct page *old_page,unsigned int flags) {struct page *new_page =NULL ;spinlock_t *ptl =NULL ;
@@ -2068,6 +2075,7 @@ static int wp_page_copy(struct mm_struct *mm, struct vm_area_struct *vma,const unsigned long mmun_start = address &PAGE_MASK ;/* For mmu_notifiers */ const unsigned long mmun_end = mmun_start +PAGE_SIZE ;/* For mmu_notifiers */ struct mem_cgroup *memcg; +bool do_uffd = false;if (unlikely (anon_vma_prepare (vma))) goto oom;
@@ -2105,7 +2113,15 @@ static int wp_page_copy(struct mm_struct *mm, struct vm_area_struct *vma, }flush_cache_page (vma, address,pte_pfn (orig_pte)); entry =mk_pte (new_page, vma->vm_page_prot); - entry =maybe_mkwrite (pte_mkdirty (entry), vma); +/* +* there is no way to know if we should do writeprotect here, +* force a writeprotect +*/ +if (userfaultfd_wp (vma)) { + entry =pte_mkdirty (entry); + do_uffd = true; + }else + entry =maybe_mkwrite (pte_mkdirty (entry), vma);/* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition
@@ -2173,6 +2189,8 @@ static int wp_page_copy(struct mm_struct *mm, struct vm_area_struct *vma, }page_cache_release (old_page); } +if (do_uffd) +return handle_userfault (vma, address, flags,VM_UFFD_WP );return page_copied ?VM_FAULT_WRITE :0 ; oom_free_new:page_cache_release (new_page);
@@ -2189,7 +2207,7 @@ static int wp_page_copy(struct mm_struct *mm, struct vm_area_struct *vma,static int wp_pfn_shared (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *page_table,spinlock_t *ptl,pte_t orig_pte, -pmd_t *pmd) +pmd_t *pmd,unsigned int flags) {if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) { struct vm_fault vmf = {
@@ -2215,13 +2233,13 @@ static int wp_pfn_shared(struct mm_struct *mm, } }return wp_page_reuse (mm, vma, address, page_table, ptl, orig_pte, -NULL ,0 ,0 ); +NULL ,0 ,0 , flags); }static int wp_page_shared (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *page_table,pmd_t *pmd,spinlock_t *ptl,pte_t orig_pte, -struct page *old_page) +struct page *old_page,unsigned int flags)__releases (ptl) {int page_mkwrite =0 ;
@@ -2261,7 +2279,7 @@ static int wp_page_shared(struct mm_struct *mm, struct vm_area_struct *vma, }return wp_page_reuse (mm, vma, address, page_table, ptl, - orig_pte, old_page, page_mkwrite,1 ); + orig_pte, old_page, page_mkwrite,1 , flags); }/*
@@ -2284,7 +2302,7 @@ static int wp_page_shared(struct mm_struct *mm, struct vm_area_struct *vma,*/ static int do_wp_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *page_table,pmd_t *pmd, -spinlock_t *ptl,pte_t orig_pte) +spinlock_t *ptl,pte_t orig_pte,unsigned int flags)__releases (ptl) {struct page *old_page;
@@ -2301,11 +2319,11 @@ static int do_wp_page(struct mm_struct *mm, struct vm_area_struct *vma,if ((vma->vm_flags & (VM_WRITE |VM_SHARED )) == (VM_WRITE |VM_SHARED ))return wp_pfn_shared (mm, vma, address, page_table, ptl, - orig_pte, pmd); + orig_pte, pmd, flags);pte_unmap_unlock (page_table, ptl);return wp_page_copy (mm, vma, address, page_table, pmd, - orig_pte, old_page); + orig_pte, old_page, flags); }/*
@@ -2336,13 +2354,13 @@ static int do_wp_page(struct mm_struct *mm, struct vm_area_struct *vma,page_move_anon_rmap (old_page, vma, address);unlock_page (old_page);return wp_page_reuse (mm, vma, address, page_table, ptl, - orig_pte, old_page,0 ,0 ); + orig_pte, old_page,0 ,0 , flags); }unlock_page (old_page); }else if (unlikely ((vma->vm_flags & (VM_WRITE |VM_SHARED )) == (VM_WRITE |VM_SHARED ))) {return wp_page_shared (mm, vma, address, page_table, pmd, - ptl, orig_pte, old_page); + ptl, orig_pte, old_page, flags); }/*
@@ -2352,7 +2370,7 @@ static int do_wp_page(struct mm_struct *mm, struct vm_area_struct *vma,pte_unmap_unlock (page_table, ptl);return wp_page_copy (mm, vma, address, page_table, pmd, - orig_pte, old_page); + orig_pte, old_page, flags); } static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2455,6 +2473,7 @@ static int do_swap_page(struct mm_struct *mm, struct vm_area_struct *vma,int locked;int exclusive =0 ;int ret =0 ; +bool do_uffd = false;if (!pte_unmap_same (mm, pmd, page_table, orig_pte)) goto out;
@@ -2559,7 +2578,15 @@ static int do_swap_page(struct mm_struct *mm, struct vm_area_struct *vma,dec_mm_counter_fast (mm,MM_SWAPENTS ); pte =mk_pte (page, vma->vm_page_prot);if ((flags &FAULT_FLAG_WRITE ) &&reuse_swap_page (page)) { - pte =maybe_mkwrite (pte_mkdirty (pte), vma); +/* +* there is no way to know if we should do writeprotect here, +* force a writeprotect +*/ +if (userfaultfd_wp (vma)) { + pte =pte_mkdirty (pte); + do_uffd = true; + }else + pte =maybe_mkwrite (pte_mkdirty (pte), vma); flags &= ~FAULT_FLAG_WRITE ; ret |=VM_FAULT_WRITE ; exclusive =1 ;
@@ -2595,7 +2622,8 @@ static int do_swap_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (flags &FAULT_FLAG_WRITE ) { - ret |=do_wp_page (mm, vma, address, page_table, pmd, ptl, pte); + ret |=do_wp_page (mm, vma, address, page_table, pmd, ptl, + pte, flags);if (ret &VM_FAULT_ERROR ) ret &=VM_FAULT_ERROR ; goto out;
@@ -2603,6 +2631,10 @@ static int do_swap_page(struct mm_struct *mm, struct vm_area_struct *vma,/* No need to invalidate - it was non-present before */ update_mmu_cache (vma, address, page_table); +if (do_uffd) { +pte_unmap_unlock (page_table, ptl); +return handle_userfault (vma, address, flags,VM_UFFD_WP ); + } unlock:pte_unmap_unlock (page_table, ptl); out:
@@ -3309,7 +3341,7 @@ static int handle_pte_fault(struct mm_struct *mm,if (flags &FAULT_FLAG_WRITE ) {if (!pte_write (entry))return do_wp_page (mm, vma, address, - pte, pmd, ptl, entry); + pte, pmd, ptl, entry, flags); entry =pte_mkdirty (entry); } entry =pte_mkyoung (entry);
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 6/8] userfaultfd: hook userfault handler to write protection fault
Quoted 37 lines · Shaohua Li
There are several cases write protection fault happens. It could be a write to zero page, swaped page or userfault write protected page. When the fault happens, there is no way to know if userfault write protect the page before. Here we just blindly issue a userfault notification for vma with
VM_UFFD_WPregardless if app write protects it yet. Application should be ready to handle such wp fault.Cc Andrea Arcangeli <[email protected]>Cc Pavel Emelyanov <[email protected]>Cc Rik van Riel <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Signed-off-by Shaohua Li <[email protected]>--- mm/memory.c | 66 +++++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 17 deletions(-)mm/memory.c +2
-1@@ -1994,10 +1994,11 @@ static inline int wp_page_reuse(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, pte_t *page_table, spinlock_t *ptl, pte_t orig_pte, struct page *page, int page_mkwrite, - int dirty_shared) + int dirty_shared, unsigned int flags) __releases(ptl) { pte_t entry; +bool do_uffd = false;/* * Clear the pages cpupid information as the existing * information potentially belongs to a now completely
mm/memory.c static inline int wp_page_reuse(struct mm_struct *mm, · line 2013
Quoted 6 lines
mm/memory.c +2
-1@@ -2008,10 +2009,16 @@ static inline int wp_page_reuse(struct mm_struct *mm,flush_cache_page (vma, address,pte_pfn (orig_pte)); entry =pte_mkyoung (orig_pte); - entry =maybe_mkwrite (pte_mkdirty (entry), vma); +if (userfaultfd_wp (vma) && page) { + entry =pte_mkdirty (entry);
Why do you pte_mkdirty() it makes no sense to me unless i am missing something. In fact, IIRC, userfaultd is only concerning private anonymous vma so you should only need to modify 3 places. do_anonymous_page(), do_swap_page() and do_wp_page()
You also want to hook in wp_huge_pmd() and __do_huge_pmd_anonymous_page() to properly cover THP.
So i think you need to simplify this patch and make sure you handle THP properly.
Cheers,
Jerome
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 7/8 userfaultfd: fault try one more time
In reply to Shaohua LiFor a swapin memory write fault, fault handler already retry once to read the page in. userfaultfd can't do the retry again and fail. Give another retry for userfaultfd in such case. gup isn't fixed yet, so will return -EBUSY.
--- arch/alpha/mm/fault.c | 8 +++++++- arch/arc/mm/fault.c | 8 +++++++- arch/arm/mm/fault.c | 8 +++++++- arch/arm64/mm/fault.c | 8 +++++++- arch/avr32/mm/fault.c | 8 +++++++- arch/cris/mm/fault.c | 8 +++++++- arch/hexagon/mm/vm_fault.c | 8 +++++++- arch/ia64/mm/fault.c | 8 +++++++- arch/m68k/mm/fault.c | 8 +++++++- arch/metag/mm/fault.c | 8 +++++++- arch/microblaze/mm/fault.c | 8 +++++++- arch/mips/mm/fault.c | 8 +++++++- arch/mn10300/mm/fault.c | 8 +++++++- arch/nios2/mm/fault.c | 8 +++++++- arch/openrisc/mm/fault.c | 8 +++++++- arch/parisc/mm/fault.c | 8 +++++++- arch/powerpc/mm/fault.c | 8 +++++++- arch/s390/mm/fault.c | 9 ++++++++- arch/sh/mm/fault.c | 8 +++++++- arch/sparc/mm/fault_32.c | 8 +++++++- arch/sparc/mm/fault_64.c | 8 +++++++- arch/tile/mm/fault.c | 8 +++++++- arch/um/kernel/trap.c | 8 +++++++- arch/unicore32/mm/fault.c | 8 +++++++- arch/x86/mm/fault.c | 9 ++++++++- arch/xtensa/mm/fault.c | 8 +++++++- fs/userfaultfd.c | 5 +++-- include/linux/mm.h | 3 ++- 28 files changed, 189 insertions(+), 29 deletions(-)
arch/alpha/mm/fault.c +7 -1
@@ -88,7 +88,8 @@ do_page_fault(unsigned long address, unsigned long mmcsr,const struct exception_table_entry *fixup;int fault, si_code =SEGV_MAPERR ;siginfo_t info; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;/* As of EV6, a load into $31/$f31 is a prefetch, and never faults (or is suppressed by the PALcode). Support that for older CPUs
@@ -178,6 +179,11 @@ do_page_fault(unsigned long address, unsigned long mmcsr, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);
arch/arc/mm/fault.c +7 -1
@@ -68,7 +68,8 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)siginfo_t info;int fault, ret;int write = regs->ecr_cause &ECR_C_PROTV_STORE ;/* ST/EX */ -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;/* * We fault-in kernel-space virtual memory on-demand. The
@@ -168,6 +169,11 @@ void do_page_fault(unsigned long address, struct pt_regs *regs) goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }/* Fault Handled Gracefully */ up_read (&mm->mmap_sem);
arch/arm/mm/fault.c +7 -1
@@ -260,7 +260,8 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)struct task_struct *tsk;struct mm_struct *mm;int fault, sig, code; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;if (notify_page_fault (regs, fsr))return 0 ;
@@ -342,6 +343,11 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) goto retry; } } +if (!(fault &VM_FAULT_ERROR ) && (flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);
arch/arm64/mm/fault.c +7 -1
@@ -199,7 +199,8 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,struct mm_struct *mm;int fault, sig, code;unsigned long vm_flags =VM_READ |VM_WRITE |VM_EXEC ; -unsigned int mm_flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int mm_flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; tsk = current; mm = tsk->mm;
@@ -291,6 +292,11 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, goto retry; } } +if ((mm_flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + mm_flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);
arch/avr32/mm/fault.c +7 -1
@@ -64,7 +64,8 @@ asmlinkage void do_page_fault(unsigned long ecr, struct pt_regs *regs)long signr;int code;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;if (notify_page_fault (regs, ecr))return ;
@@ -166,6 +167,11 @@ asmlinkage void do_page_fault(unsigned long ecr, struct pt_regs *regs) goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/cris/mm/fault.c +7 -1
@@ -58,7 +58,8 @@ do_page_fault(unsigned long address, struct pt_regs *regs,struct vm_area_struct * vma;siginfo_t info;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; D(printk(KERN_DEBUG "Page fault for %lX on %X at %lX, prot %d write %d\n",
@@ -201,6 +202,11 @@ do_page_fault(unsigned long address, struct pt_regs *regs, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/hexagon/mm/vm_fault.c +7 -1
@@ -53,7 +53,8 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)int si_code =SEGV_MAPERR ;int fault;const struct exception_table_entry *fixup; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;/* * If we're in an interrupt or have no user context,
@@ -119,6 +120,11 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs) goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/ia64/mm/fault.c +7 -1
@@ -85,7 +85,8 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *restruct siginfo si;unsigned long mask;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; mask = ((((isr >>IA64_ISR_X_BIT ) &1UL ) <<VM_EXEC_BIT ) | (((isr >>IA64_ISR_W_BIT ) &1UL ) <<VM_WRITE_BIT ));
@@ -198,6 +199,11 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/m68k/mm/fault.c +7 -1
@@ -72,7 +72,8 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,struct mm_struct *mm = current->mm;struct vm_area_struct * vma;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;pr_debug ("do page fault:\nregs->sr=%#x, regs->pc=%#lx, address=%#lx, %ld, %p\n" , regs->sr, regs->pc, address, error_code, mm ? mm->pgd :NULL );
@@ -177,6 +178,11 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return 0 ;
arch/metag/mm/fault.c +7 -1
@@ -53,7 +53,8 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,struct vm_area_struct *vma, *prev_vma;siginfo_t info;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; tsk = current;
@@ -165,6 +166,11 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return 0 ;
arch/microblaze/mm/fault.c +7 -1
@@ -92,7 +92,8 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,int code =SEGV_MAPERR ;int is_write = error_code &ESR_S ;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; regs->ear = address; regs->esr = error_code;
@@ -249,6 +250,11 @@ void do_page_fault(struct pt_regs *regs, unsigned long address, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);
arch/mips/mm/fault.c +7 -1
@@ -45,7 +45,8 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,const int field =sizeof (unsigned long ) *2 ;siginfo_t info;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 10);
@@ -191,6 +192,11 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/mn10300/mm/fault.c +7 -1
@@ -124,7 +124,8 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long fault_code,unsigned long page;siginfo_t info;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;#ifdef CONFIG_GDBSTUB /* handle GDB stub causing a fault */
@@ -284,6 +285,11 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long fault_code, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/nios2/mm/fault.c +7 -1
@@ -47,7 +47,8 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,struct mm_struct *mm = tsk->mm;int code =SEGV_MAPERR ;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; cause >>=2 ;
@@ -171,6 +172,11 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/openrisc/mm/fault.c +7 -1
@@ -54,7 +54,8 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,struct vm_area_struct *vma;siginfo_t info;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; tsk = current;
@@ -196,6 +197,11 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/parisc/mm/fault.c +7 -1
@@ -215,7 +215,8 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,if (!mm) goto no_context; - flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; + flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;if (user_mode (regs)) flags |=FAULT_FLAG_USER ;
@@ -279,6 +280,11 @@ void do_page_fault(struct pt_regs *regs, unsigned long code, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/powerpc/mm/fault.c +7 -1
@@ -211,7 +211,8 @@ int __kprobes do_page_fault(struct pt_regs *regs, unsigned long address,enum ctx_state prev_state =exception_enter ();struct vm_area_struct * vma;struct mm_struct *mm = current->mm; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;int code =SEGV_MAPERR ;int is_write =0 ;int trap =TRAP (regs);
@@ -474,6 +475,11 @@ int __kprobes do_page_fault(struct pt_regs *regs, unsigned long address, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem); goto bail;
arch/s390/mm/fault.c +8 -1
@@ -405,7 +405,8 @@ static inline int do_exception(struct pt_regs *regs, int access) address = trans_exc_code & __FAIL_ADDR_MASK;perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); - flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; + flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;if (user_mode (regs)) flags |=FAULT_FLAG_USER ;if (access ==VM_WRITE || (trans_exc_code & store_indication) ==0x400 )
@@ -498,6 +499,12 @@ static inline int do_exception(struct pt_regs *regs, int access) goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; +down_read (&mm->mmap_sem); + goto retry; + }#ifdef CONFIG_PGSTE if (gmap) { address = __gmap_link(gmap, current->thread.gmap_addr,
arch/sh/mm/fault.c +7 -1
@@ -403,7 +403,8 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,struct mm_struct *mm;struct vm_area_struct * vma;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; tsk = current; mm = tsk->mm;
@@ -515,6 +516,11 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem); }
arch/sparc/mm/fault_32.c +7 -1
@@ -174,7 +174,8 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,unsigned long g2;int from_user = !(regs->psr &PSR_PS );int fault, code; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ;if (text_fault) address = regs->pc;
@@ -278,6 +279,11 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);return ;
arch/sparc/mm/fault_64.c +7 -1
@@ -287,7 +287,8 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)unsigned int insn =0 ;int si_code, fault_code, fault;unsigned long address, mm_rss; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; fault_code =get_thread_fault_code ();
@@ -476,6 +477,11 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem); mm_rss =get_mm_rss (mm);
arch/tile/mm/fault.c +7 -1
@@ -278,7 +278,8 @@ static int handle_page_fault(struct pt_regs *regs,if (!is_page_fault) write =1 ; - flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; + flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; is_kernel_mode = !user_mode (regs);
@@ -466,6 +467,11 @@ static int handle_page_fault(struct pt_regs *regs, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }#if CHIP_HAS_TILE_DMA ()/* If this was a DMA TLB fault, restart the DMA engine. */
arch/um/kernel/trap.c +7 -1
@@ -31,7 +31,8 @@ int handle_page_fault(unsigned long address, unsigned long ip,pmd_t *pmd;pte_t *pte;int err = -EFAULT ; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; *code_out =SEGV_MAPERR ;
@@ -101,6 +102,11 @@ int handle_page_fault(unsigned long address, unsigned long ip, goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + } pgd =pgd_offset (mm, address); pud =pud_offset (pgd, address);
arch/unicore32/mm/fault.c +7 -1
@@ -209,7 +209,8 @@ static int do_pf(unsigned long addr, unsigned int fsr, struct pt_regs *regs)struct task_struct *tsk;struct mm_struct *mm;int fault, sig, code; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; tsk = current; mm = tsk->mm;
@@ -272,6 +273,11 @@ static int do_pf(unsigned long addr, unsigned int fsr, struct pt_regs *regs) goto retry; } } +if (!(fault &VM_FAULT_ERROR ) && (flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);
arch/x86/mm/fault.c +8 -1
@@ -1062,7 +1062,8 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,struct task_struct *tsk;struct mm_struct *mm;int fault, major =0 ; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; tsk = current; mm = tsk->mm;
@@ -1251,6 +1252,12 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,if (!fatal_signal_pending (tsk)) goto retry; } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; +if (!fatal_signal_pending (tsk)) + goto retry; + }/* User mode? Just return to handle the fatal exception */ if (flags &FAULT_FLAG_USER )
arch/xtensa/mm/fault.c +7 -1
@@ -45,7 +45,8 @@ void do_page_fault(struct pt_regs *regs)int is_write, is_exec;int fault; -unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ; +unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE | +FAULT_FLAG_ALLOW_UFFD_RETRY ; info.si_code =SEGV_MAPERR ;
@@ -141,6 +142,11 @@ void do_page_fault(struct pt_regs *regs) goto retry; } } +if ((flags &FAULT_FLAG_ALLOW_UFFD_RETRY ) && + (fault &VM_FAULT_UFFD_RETRY )) { + flags &= ~FAULT_FLAG_ALLOW_UFFD_RETRY ; + goto retry; + }up_read (&mm->mmap_sem);perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address);
fs/userfaultfd.c +3 -2
@@ -298,7 +298,8 @@ int handle_userfault(struct vm_area_struct *vma, unsigned long address,* without first stopping userland access to the memory. For * VM_UFFD_MISSING userfaults this is enough for now. */ -if (unlikely (!(flags &FAULT_FLAG_ALLOW_RETRY ))) { +if (unlikely (!(flags & (FAULT_FLAG_ALLOW_RETRY | +FAULT_FLAG_ALLOW_UFFD_RETRY )))) {/* * Validate the invariant that nowait must allow retry * to be sure not to return SIGBUS erroneously on
@@ -357,7 +358,7 @@ int handle_userfault(struct vm_area_struct *vma, unsigned long address, !fatal_signal_pending(current)))) {wake_up_poll (&ctx->fd_wqh,POLLIN );schedule (); - ret |=VM_FAULT_MAJOR ; + ret |=VM_FAULT_MAJOR |VM_FAULT_UFFD_RETRY ; }__set_current_state (TASK_RUNNING );
include/linux/mm.h +2 -1
@@ -219,6 +219,7 @@ extern pgprot_t protection_map[16];#define FAULT_FLAG_KILLABLE 0x10/* The fault task is in SIGKILL killable region */ #define FAULT_FLAG_TRIED 0x20/* Second try */ #define FAULT_FLAG_USER 0x40/* The fault originated in userspace */ +#define FAULT_FLAG_ALLOW_UFFD_RETRY 0x80/* userfault retry */ /* * vm_fault is filled by the the pagefault handler and passed to the vma's
@@ -1027,7 +1028,7 @@ static inline void clear_page_pfmemalloc(struct page *page)#define VM_FAULT_HWPOISON 0x0010/* Hit poisoned small page */ #define VM_FAULT_HWPOISON_LARGE 0x0020/* Hit poisoned large page. Index encoded in upper bits */ #define VM_FAULT_SIGSEGV 0x0040 - +#define VM_FAULT_UFFD_RETRY 0x0080#define VM_FAULT_NOPAGE 0x0100/* ->fault installed the pte, not return page */ #define VM_FAULT_LOCKED 0x0200/* ->fault locked the returned page */ #define VM_FAULT_RETRY 0x0400/* ->fault blocked, must retry */
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 7/8] userfaultfd: fault try one more time
Quoted 4 lines · Shaohua Li
For a swapin memory write fault, fault handler already retry once to read the page in. userfaultfd can't do the retry again and fail. Give another retry for userfaultfd in such case. gup isn't fixed yet, so will return
-EBUSY.
This whole patch make me nervous. I do not see the point in it. So on page fault in first pass you have the RETRY flag set and you can either return VM_FAULT_RETRY because (1) lock_page_or_retry() in do_swap_page() or because (2) handle_userfault().
In second case, on retry you already have a valid read only pte so you go directly to do_wp_page() and this is properly handle by current handle_userfault() code. So it does not make sense to add complexity for that case.
You seem to hint that you are doing this for the first case (1) but even for that one it does not make sense. So if we fail to lock the page it is because someone else is doing something with that page and most likely it is related to the userfaultfd already (like another thread took the fault and is doing all the steps you need). So you just want a regular retry, ie do_swap_page() return retry and on retry it is likely that everything is already all good. If not that it takes the slow painful wait code path.
I genuinely do not see what benefit and reasons there is to this new special usefaultfd retry flag.
Cheers,
Jerome
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 8/8 userfaultfd: enabled write protection in userfaultfd API
In reply to Shaohua LiNow it's safe to enable write protection in userfaultfd API
--- include/uapi/linux/userfaultfd.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
include/uapi/linux/userfaultfd.h +4 -3
@@ -17,7 +17,7 @@* #define UFFD_API_FEATURES (UFFD_FEATURE_PAGEFAULT_FLAG_WP | \ * UFFD_FEATURE_EVENT_FORK) */ -#define UFFD_API_FEATURES (0) +#define UFFD_API_FEATURES (UFFD_FEATURE_PAGEFAULT_FLAG_WP)#define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \
@@ -25,7 +25,8 @@#define UFFD_API_RANGE_IOCTLS \ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY | \ - (__u64)1 << _UFFDIO_ZEROPAGE) + (__u64)1 << _UFFDIO_ZEROPAGE | \ + (__u64)1 << _UFFDIO_WRITEPROTECT)/* * Valid ioctl command number range with this API is from 0x00 to
@@ -108,8 +109,8 @@ struct uffdio_api {* are to be considered implicitly always enabled in all kernels as * long as the uffdio_api.api requested matches UFFD_API. */ -#if 0 /* not available yet */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) +#if 0 /* not available yet */ #define UFFD_FEATURE_EVENT_FORK (1<<1)#endif __u64 features;
Signature · 7 lines
2.4.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · Shaohua Li
Hi,
There is plan to support write protect fault into userfaultfd before, but it's not implemented yet. I'm working on a library to support different types of buffer like compressed buffer and file buffer, something like a page cache implementation in userspace. The buffer enables userfaultfd and does something like decompression in userfault handler. When memory size exceeds a threshold, madvise is used to reclaim memory. The problem is data can be corrupted in reclaim without memory protection support.
For example, in the compressed buffer case, reclaim does:
1. compress memory range and store compressed data elsewhere
2. madvise the memory range
Quoted 10 lines
But if the memory is changed before 2, new change is lost. memory write protection can solve the issue. With it, the reclaim does:
1. write protect memory range
2. compress memory range and store compressed data elsewhere
3. madvise the memory range
4. undo write protect memory range and wakeup tasks waiting in write protect fault.
If a task changes memory before 3, write protect fault will be triggered. we can put the task into sleep till step 4 runs for example. In this way memory changes will not be lost.
While i understand the whole concept of write protection while doing compression. I do not see valid usecase for this. Inside the kernel we already have thing like zswap that already does what you seem to want to do (ie compress memory range and transparently uncompress it on next CPU access).
I fail to see a usecase where we would realy would like to do this in userspace.
Quoted 22 lines
This patch set add write protect support for userfaultfd. One issue is write protect fault can happen even without enabling write protect in userfault. For example, a write to address backed by zero page. There is no way to distinguish if this is a write protect fault expected by userfault. This patch just blindly triggers write protect fault to userfault if corresponding vma enables
VM_UFFD_WP. Application should be prepared to handle such write protect fault.Thanks,
ShaohuaShaohua Li (8):
userfaultfd: add helper for writeprotect check
userfaultfd: support write protection for userfault vma range
userfaultfd: expose writeprotect API to ioctl
userfaultfd: allow userfaultfd register success with writeprotection
userfaultfd: undo write proctection in unregister
userfaultfd: hook userfault handler to write protection fault
userfaultfd: fault try one more time
userfaultfd: enabled write protection in userfaultfd API
Quoted 1 line
From organization point of view, i would put the "expose writeprotect API to ioctl"
as the last patch in the serie after all the plumbing is done. This would make "enabled write protection in userfaultfd API" useless and avoid akward changes in some of the others patches where you add commented/disabled code.
Also you want to handle GUP, like you want the write protection to fails if there is GUP and you want GUP to force breaking write protection, otherwise this will be broken if anyone mix it with something that trigger GUP.
Cheers,
Jerome
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>