userfaultfd-wp: Support shmem and hugetlbfs
Subsystems (18)
Peter Xu
reviewerv7 cover userfaultfd-wp: Support shmem and hugetlbfs
This is v7 of the series to add shmem+hugetlbfs support for userfaultfd write protection. It is based on linux-next tag next-20220301.
The whole tree can be found here for testing:
https://github.com/xzpeter/linux/tree/uffd-wp-shmem-hugetlbfs
One tweak needed is to replace Nadav's v2 patch "userfaultfd: provide unmasked address on page-fault" with v3 to unbreak hugetlb in general.
There aren't a lot of changes comparing to v6, majorly rebase and retest to make sure nothing breaks. Meanwhile should have addressed comments from Alistair.
v7 changelog:
- Rebased to next-20220301
- Renamed s/is_pte_marker_uffd_wp/pte_marker_uffd_wp/, add another helper pte_marker_entry_uffd_wp as suggested to operate on swp_entry_t [Alistair]
- Drop pte_unmap_same() in pte_marker_handle_uffd_wp() [Alistair]
- In finish_fault(), init vmf->orig_pte with pte_clear(), because some pte_none() ptes are not really all zeros, e.g. xtensa and s390 [Alistair]
Previous versions:
RFC: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/ v2: https://lore.kernel.org/lkml/[email protected]/ v3: https://lore.kernel.org/lkml/[email protected]/ v4: https://lore.kernel.org/lkml/[email protected]/ v5: https://lore.kernel.org/lkml/[email protected]/ v6: https://lore.kernel.org/lkml/[email protected]/
Overview
========
Userfaultfd-wp anonymous support was merged two years ago. There're quite a few applications that started to leverage this capability either to take snapshots for user-app memory, or use it for full user controled swapping.
This series tries to complete the feature for uffd-wp so as to cover all the RAM-based memory types. So far uffd-wp is the only missing piece of the rest features (uffd-missing & uffd-minor mode).
One major reason to do so is that anonymous pages are sometimes not satisfying the need of applications, and there're growing users of either shmem and hugetlbfs for either sharing purpose (e.g., sharing guest mem between hypervisor process and device emulation process, shmem local live migration for upgrades), or for performance on tlb hits.
All these mean that if a uffd-wp app wants to switch to any of the memory types, it'll stop working. I think it's worthwhile to have the kernel to cover all these aspects.
This series chose to protect pages in pte level not page level.
One major reason is safety. I have no idea how we could make it safe if any of the uffd-privileged app can wr-protect a page that any other application can use. It means this app can block any process potentially for any time it wants.
The other reason is that it aligns very well with not only the anonymous uffd-wp solution, but also uffd as a whole. For example, userfaultfd is implemented fundamentally based on VMAs. We set flags to VMAs showing the status of uffd tracking. For another per-page based protection solution, it'll be crossing the fundation line on VMA-based, and it could simply be too far away already from what's called userfaultfd.
PTE markers
===========
The patchset is based on the idea called PTE markers. It was discussed in one of the mm alignment sessions, proposed starting from v6, and this is the 2nd version of it using PTE marker idea.
PTE marker is a new type of swap entry that is ony applicable to file backed memories like shmem and hugetlbfs. It's used to persist some pte-level information even if the original present ptes in pgtable are zapped.
Logically pte markers can store more than uffd-wp information, but so far only one bit is used for uffd-wp purpose. When the pte marker is installed with uffd-wp bit set, it means this pte is wr-protected by uffd.
It solves the problem on e.g. file-backed memory mapped ptes got zapped due to any reason (e.g. thp split, or swapped out), we can still keep the wr-protect information in the ptes. Then when the page fault triggers again, we'll know this pte is wr-protected so we can treat the pte the same as a normal uffd wr-protected pte.
The extra information is encoded into the swap entry, or swp_offset to be explicit, with the swp_type being PTE_MARKER. So far uffd-wp only uses one bit out of the swap entry, the rest bits of swp_offset are still reserved for other purposes.
There're two configs to enable/disable PTE markers:
CONFIG_PTE_MARKER
CONFIG_PTE_MARKER_UFFD_WP
We can set !PTE_MARKER to completely disable all the PTE markers, along with uffd-wp support. I made two config so we can also enable PTE marker but disable uffd-wp file-backed for other purposes. At the end of current series, I'll enable CONFIG_PTE_MARKER by default, but that patch is standalone and if anyone worries about having it by default, we can also consider turn it off by dropping that oneliner patch. So far I don't see a huge risk of doing so, so I kept that patch.
In most cases, PTE markers should be treated as none ptes. It is because that unlike most of the other swap entry types, there's no PFN or block offset information encoded into PTE markers but some extra well-defined bits showing the status of the pte. These bits should only be used as extra data when servicing an upcoming page fault, and then we behave as if it's a none pte.
I did spend a lot of time observing all the pte_none() users this time. It is indeed a challenge because there're a lot, and I hope I didn't miss a single of them when we should take care of pte markers. Luckily, I don't think it'll need to be considered in many cases, for example: boot code, arch code (especially non-x86), kernel-only page handlings (e.g. CPA), or device driver codes when we're tackling with pure PFN mappings.
I introduced pte_none_mostly() in this series when we need to handle pte markers the same as none pte, the "mostly" is the other way to write "either none pte or a pte marker".
I didn't replace pte_none() to cover pte markers for below reasons:
- Very rare case of pte_none() callers will handle pte markers. E.g., all the kernel pages do not require knowledge of pte markers. So we don't pollute the major use cases.
- Unconditionally change pte_none() semantics could confuse people, because pte_none() existed for so long a time.
- Unconditionally change pte_none() semantics could make pte_none() slower even if in many cases pte markers do not exist.
- There're cases where we'd like to handle pte markers differntly from pte_none(), so a full replace is also impossible. E.g. khugepaged should still treat pte markers as normal swap ptes rather than none ptes, because pte markers will always need a fault-in to merge the marker with a valid pte. Or the smap code will need to parse PTE markers not none ptes.
Patch Layout
============
Introducing PTE marker and uffd-wp bit in PTE marker:
mm: Introduce PTE_MARKER swap entry
mm: Teach core mm about pte markers
mm: Check against orig_pte for finish_fault()
mm/uffd: PTE_MARKER_UFFD_WP
Adding support for shmem uffd-wp:
mm/shmem: Take care of UFFDIO_COPY_MODE_WP
mm/shmem: Handle uffd-wp special pte in page fault handler mm/shmem: Persist uffd-wp bit across zapping for file-backed mm/shmem: Allow uffd wr-protect none pte for file-backed mem mm/shmem: Allows file-back mem to be uffd wr-protected on thps mm/shmem: Handle uffd-wp during fork()
Adding support for hugetlbfs uffd-wp:
mm/hugetlb: Introduce huge pte version of uffd-wp helpers mm/hugetlb: Hook page faults for uffd write protection mm/hugetlb: Take care of UFFDIO_COPY_MODE_WP mm/hugetlb: Handle UFFDIO_WRITEPROTECT mm/hugetlb: Handle pte markers in page faults mm/hugetlb: Allow uffd wr-protect none ptes mm/hugetlb: Only drop uffd-wp special pte if required mm/hugetlb: Handle uffd-wp during fork()
Misc handling on the rest mm for uffd-wp file-backed:
mm/khugepaged: Don't recycle vma pgtable if uffd-wp registered mm/pagemap: Recognize uffd-wp bit for shmem/hugetlbfs
Enabling of uffd-wp on file-backed memory:
mm/uffd: Enable write protection for shmem & hugetlbfs
mm: Enable PTE markers by default selftests/uffd: Enable uffd-wp for shmem/hugetlbfs
Tests
=====
- Compile test on x86_64 and aarch64 on different configs
- Kernel selftests
- uffd-test [0]
- Umapsort [1,2] test for shmem/hugetlb, with swap on/off
Please review, thanks.
[0] https://github.com/xzpeter/clibs/tree/master/uffd-test
[1] https://github.com/xzpeter/umap-apps/tree/peter
[2] https://github.com/xzpeter/umap/tree/peter-shmem-hugetlbfs
Peter Xu (23):
mm: Introduce PTE_MARKER swap entry
mm: Teach core mm about pte markers
mm: Check against orig_pte for finish_fault() mm/uffd: PTE_MARKER_UFFD_WP
mm/shmem: Take care of UFFDIO_COPY_MODE_WP
mm/shmem: Handle uffd-wp special pte in page fault handler mm/shmem: Persist uffd-wp bit across zapping for file-backed mm/shmem: Allow uffd wr-protect none pte for file-backed mem mm/shmem: Allows file-back mem to be uffd wr-protected on thps mm/shmem: Handle uffd-wp during fork()
mm/hugetlb: Introduce huge pte version of uffd-wp helpers mm/hugetlb: Hook page faults for uffd write protection mm/hugetlb: Take care of UFFDIO_COPY_MODE_WP mm/hugetlb: Handle UFFDIO_WRITEPROTECT
mm/hugetlb: Handle pte markers in page faults mm/hugetlb: Allow uffd wr-protect none ptes mm/hugetlb: Only drop uffd-wp special pte if required mm/hugetlb: Handle uffd-wp during fork() mm/khugepaged: Don't recycle vma pgtable if uffd-wp registered mm/pagemap: Recognize uffd-wp bit for shmem/hugetlbfs mm/uffd: Enable write protection for shmem & hugetlbfs
mm: Enable PTE markers by default
selftests/uffd: Enable uffd-wp for shmem/hugetlbfs
arch/s390/include/asm/hugetlb.h | 15 ++ fs/hugetlbfs/inode.c | 15 +- fs/proc/task_mmu.c | 7 + fs/userfaultfd.c | 31 +--- include/asm-generic/hugetlb.h | 24 +++ include/linux/hugetlb.h | 27 ++-- include/linux/mm.h | 10 ++ include/linux/mm_inline.h | 43 +++++ include/linux/shmem_fs.h | 4 +- include/linux/swap.h | 15 +- include/linux/swapops.h | 79 +++++++++ include/linux/userfaultfd_k.h | 71 ++++++++ include/uapi/linux/userfaultfd.h | 10 +- mm/Kconfig | 16 ++ mm/filemap.c | 5 + mm/hmm.c | 2 +- mm/hugetlb.c | 182 ++++++++++++++++----- mm/khugepaged.c | 14 +- mm/memcontrol.c | 8 +- mm/memory.c | 196 ++++++++++++++++++++--- mm/mincore.c | 3 +- mm/mprotect.c | 75 ++++++++- mm/rmap.c | 8 + mm/shmem.c | 4 +- mm/userfaultfd.c | 54 +++++-- tools/testing/selftests/vm/userfaultfd.c | 4 +- 26 files changed, 795 insertions(+), 127 deletions(-)
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 1/23 mm: Introduce PTE_MARKER swap entry
This patch introduces a new swap entry type called PTE_MARKER. It can be installed for any pte that maps a file-backed memory when the pte is temporarily zapped, so as to maintain per-pte information.
The information that kept in the pte is called a "marker". Here we define the marker as "unsigned long" just to match pgoff_t, however it will only work if it still fits in swp_offset(), which is e.g. currently 58 bits on x86_64.
A new config CONFIG_PTE_MARKER is introduced too; it's by default off. A bunch of helpers are defined altogether to service the rest of the pte marker code.
--- include/asm-generic/hugetlb.h | 9 ++++ include/linux/swap.h | 15 ++++++- include/linux/swapops.h | 78 +++++++++++++++++++++++++++++++++++ mm/Kconfig | 7 ++++ 4 files changed, 108 insertions(+), 1 deletion(-)
include/asm-generic/hugetlb.h +9 -0
@@ -2,6 +2,9 @@#ifndef _ASM_GENERIC_HUGETLB_H#define _ASM_GENERIC_HUGETLB_H +#include <linux/swap.h> +#include <linux/swapops.h> +static inline pte_t mk_huge_pte (struct page *page,pgprot_t pgprot) {return mk_pte (page, pgprot);
@@ -80,6 +83,12 @@ static inline int huge_pte_none(pte_t pte) }#endif +/* Please refer to comments above pte_none_mostly() for the usage */ +static inline int huge_pte_none_mostly (pte_t pte) +{ +return huge_pte_none (pte) ||is_pte_marker (pte); +} +#ifndef __HAVE_ARCH_HUGE_PTE_WRPROTECTstatic inline pte_t huge_pte_wrprotect (pte_t pte) {
include/linux/swap.h +14 -1
@@ -55,6 +55,19 @@ static inline int current_is_kswapd(void)* actions on faults. */ +/* +* PTE markers are used to persist information onto PTEs that are mapped with +* file-backed memories. As its name "PTE" hints, it should only be applied to +* the leaves of pgtables. +*/ +#ifdef CONFIG_PTE_MARKER +#define SWP_PTE_MARKER_NUM 1 +#define SWP_PTE_MARKER (MAX_SWAPFILES + SWP_HWPOISON_NUM + \ + SWP_MIGRATION_NUM + SWP_DEVICE_NUM) +#else +#define SWP_PTE_MARKER_NUM 0 +#endif +/* * Unaddressable device memory support. See include/linux/hmm.h and * Documentation/vm/hmm.rst. Short description is we need struct pages for
@@ -100,7 +113,7 @@ static inline int current_is_kswapd(void)#define MAX_SWAPFILES \ ((1 << MAX_SWAPFILES_SHIFT) - SWP_DEVICE_NUM - \ - SWP_MIGRATION_NUM - SWP_HWPOISON_NUM) + SWP_MIGRATION_NUM - SWP_HWPOISON_NUM - SWP_PTE_MARKER_NUM)/* * Magic header for a swap area. The first part of the union is
include/linux/swapops.h +78 -0
84 lines not on this page · raw message
mm/Kconfig +7 -0
@@ -900,6 +900,13 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. +config PTE_MARKER + def_bool n + bool "Marker PTEs support" + + help + Allows to create marker PTEs for file-backed memory. + source "mm/damon/Kconfig" endmenu
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 2/23 mm: Teach core mm about pte markers
In reply to Peter XuThis patch still does not use pte marker in any way, however it teaches the core mm about the pte marker idea.
For example, handle_pte_marker() is introduced that will parse and handle all the pte marker faults.
Many of the places are more about commenting it up - so that we know there's the possibility of pte marker showing up, and why we don't need special code for the cases.
--- fs/userfaultfd.c | 10 ++++++---- mm/filemap.c | 5 +++++ mm/hmm.c | 2 +- mm/memcontrol.c | 8 ++++++-- mm/memory.c | 23 +++++++++++++++++++++++ mm/mincore.c | 3 ++- mm/mprotect.c | 3 +++ 7 files changed, 46 insertions(+), 8 deletions(-)
fs/userfaultfd.c +6 -4
@@ -249,9 +249,10 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,/* * Lockless access: we're in a wait_event so it's ok if it -* changes under us. +* changes under us. PTE markers should be handled the same as none +* ptes here. */ -if (huge_pte_none (pte)) +if (huge_pte_none_mostly (pte)) ret = true;if (!huge_pte_write (pte) && (reason &VM_UFFD_WP )) ret = true;
@@ -330,9 +331,10 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, pte =pte_offset_map (pmd, address);/* * Lockless access: we're in a wait_event so it's ok if it -* changes under us. +* changes under us. PTE markers should be handled the same as none +* ptes here. */ -if (pte_none (*pte)) +if (pte_none_mostly (*pte)) ret = true;if (!pte_write (*pte) && (reason &VM_UFFD_WP )) ret = true;
mm/filemap.c +5 -0
@@ -3379,6 +3379,11 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, vmf->pte += xas.xa_index - last_pgoff; last_pgoff = xas.xa_index; +/* +* NOTE: If there're PTE markers, we'll leave them to be +* handled in the specific fault path, and it'll prohibit the +* fault-around logic. +*/ if (!pte_none (*vmf->pte)) goto unlock;
mm/hmm.c +1 -1
@@ -239,7 +239,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,pte_t pte = *ptep;uint64_t pfn_req_flags = *hmm_pfn; -if (pte_none (pte)) { +if (pte_none_mostly (pte)) { required_fault =hmm_pte_need_fault (hmm_vma_walk, pfn_req_flags,0 );if (required_fault)
mm/memcontrol.c +6 -2
@@ -5636,10 +5636,14 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,if (pte_present (ptent)) page =mc_handle_present_pte (vma, addr, ptent); +else if (pte_none_mostly (ptent)) +/* +* PTE markers should be treated as a none pte here, separated +* from other swap handling below. +*/ + page =mc_handle_file_pte (vma, addr, ptent);else if (is_swap_pte (ptent)) page =mc_handle_swap_pte (vma, ptent, &ent); -else if (pte_none (ptent)) - page =mc_handle_file_pte (vma, addr, ptent);if (!page && !ent.val)return ret;
mm/memory.c +23 -0
@@ -99,6 +99,8 @@ struct page *mem_map;EXPORT_SYMBOL (mem_map);#endif +static vm_fault_t do_fault (struct vm_fault *vmf); +/* * A number of key systems in x86 including ioremap() rely on the assumption * that high_memory defines the upper bound on direct map memory, then end
@@ -1419,6 +1421,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,if (!should_zap_page (details, page))continue ; rss[mm_counter (page)]--; + }else if (is_pte_marker_entry (entry)) { +/* By default, simply drop all pte markers when zap */ }else if (is_hwpoison_entry (entry)) {if (!should_zap_cows (details))continue ;
@@ -3508,6 +3512,23 @@ static inline bool should_try_to_free_swap(struct page *page,page_count (page) ==2 ; } +static vm_fault_t handle_pte_marker (struct vm_fault *vmf) +{ +swp_entry_t entry =pte_to_swp_entry (vmf->orig_pte); +unsigned long marker =pte_marker_get (entry); + +/* +* PTE markers should always be with file-backed memories, and the +* marker should never be empty. If anything weird happened, the best +* thing to do is to kill the process along with its mm. +*/ +if (WARN_ON_ONCE (vma_is_anonymous (vmf->vma) || !marker)) +return VM_FAULT_SIGBUS ; + +/* TODO: handle pte markers */ +return 0 ; +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked.
@@ -3544,6 +3565,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) ret = vmf->page->pgmap->ops->migrate_to_ram (vmf); }else if (is_hwpoison_entry (entry)) { ret =VM_FAULT_HWPOISON ; + }else if (is_pte_marker_entry (entry)) { + ret =handle_pte_marker (vmf); }else {print_bad_pte (vma, vmf->address, vmf->orig_pte,NULL ); ret =VM_FAULT_SIGBUS ;
mm/mincore.c +2 -1
@@ -121,7 +121,8 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,for (; addr != end; ptep++, addr +=PAGE_SIZE ) {pte_t pte = *ptep; -if (pte_none (pte)) +/* We need to do cache lookup too for pte markers */ +if (pte_none_mostly (pte))__mincore_unmapped_range (addr, addr +PAGE_SIZE , vma, vec);else if (pte_present (pte))
mm/mprotect.c +3 -0
@@ -184,6 +184,9 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, newpte =pte_swp_mksoft_dirty (newpte);if (pte_swp_uffd_wp (oldpte)) newpte =pte_swp_mkuffd_wp (newpte); + }else if (is_pte_marker_entry (entry)) { +/* Skip it, the same as none pte */ +continue ; }else { newpte = oldpte; }
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 3/23 mm: Check against orig_pte for finish_fault()
In reply to Peter XuWe used to check against none pte in finish_fault(), with the assumption that the orig_pte is always none pte.
This change prepares us to be able to call do_fault() on !none ptes. For example, we should allow that to happen for pte marker so that we can restore information out of the pte markers.
Let's change the "pte_none" check into detecting changes since we fetched orig_pte. One trivial thing to take care of here is, when pmd==NULL for the pgtable we may not initialize orig_pte at all in handle_pte_fault().
By default orig_pte will be all zeros however the problem is not all architectures are using all-zeros for a none pte. pte_clear() will be the right thing to use here so that we'll always have a valid orig_pte value for the whole handle_pte_fault() call.
--- mm/memory.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/memory.c +8 -1
@@ -4135,7 +4135,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf) vmf->address, &vmf->ptl); ret =0 ;/* Re-check under ptl */ -if (likely (pte_none (*vmf->pte))) +if (likely (pte_same (*vmf->pte, vmf->orig_pte)))do_set_pte (vmf, page, vmf->address);else ret =VM_FAULT_NOPAGE ;
@@ -4600,6 +4600,13 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)* concurrent faults and from rmap lookups. */ vmf->pte =NULL ; +/* +* Always initialize orig_pte. This matches with below +* code to have orig_pte to be the none pte if pte==NULL. +* This makes the rest code to be always safe to reference +* it, e.g. in finish_fault() we'll detect pte changes. +*/ +pte_clear (vmf->vma->vm_mm, vmf->address, &vmf->orig_pte); }else {/* * If a huge pmd materialized under us just retry later. Use
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 4/23 mm/uffd: PTE_MARKER_UFFD_WP
In reply to Peter XuThis patch introduces the 1st user of pte marker: the uffd-wp marker.
When the pte marker is installed with the uffd-wp bit set, it means this pte was wr-protected by uffd.
We will use this special pte to arm the ptes that got either unmapped or swapped out for a file-backed region that was previously wr-protected. This special pte could trigger a page fault just like swap entries.
This idea is greatly inspired by Hugh and Andrea in the discussion, which is referenced in the links below.
Some helpers are introduced to detect whether a swap pte is uffd wr-protected. After the pte marker introduced, one swap pte can be wr-protected in two forms:
either it is a normal swap pte and it has _PAGE_SWP_UFFD_WP set, or it's a pte marker that has PTE_MARKER_UFFD_WP set.
--- include/linux/swapops.h | 3 ++- include/linux/userfaultfd_k.h | 43 +++++++++++++++++++++++++++++++++++ mm/Kconfig | 9 ++++++++ 3 files changed, 54 insertions(+), 1 deletion(-)
include/linux/swapops.h +2 -1
@@ -249,7 +249,8 @@ static inline int is_writable_migration_entry(swp_entry_t entry)typedef unsigned long pte_marker ; -#define PTE_MARKER_MASK (0) +#define PTE_MARKER_UFFD_WP BIT(0) +#define PTE_MARKER_MASK (PTE_MARKER_UFFD_WP)#ifdef CONFIG_PTE_MARKER
include/linux/userfaultfd_k.h +43 -0
53 lines not on this page · raw message
mm/Kconfig +9 -0
@@ -907,6 +907,15 @@ config PTE_MARKER help Allows to create marker PTEs for file-backed memory. +config PTE_MARKER_UFFD_WP + bool "Marker PTEs support for userfaultfd write protection" + depends on PTE_MARKER && HAVE_ARCH_USERFAULTFD_WP + + help + Allows to create marker PTEs for userfaultfd write protection + purposes. It is required to enable userfaultfd write protection on + file-backed memory types like shmem and hugetlbfs. + source "mm/damon/Kconfig" endmenu
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 5/23 mm/shmem: Take care of UFFDIO_COPY_MODE_WP
In reply to Peter XuPass wp_copy into shmem_mfill_atomic_pte() through the stack, then apply the UFFD_WP bit properly when the UFFDIO_COPY on shmem is with UFFDIO_COPY_MODE_WP. wp_copy lands mfill_atomic_install_pte() finally.
Note: we must do pte_wrprotect() if !writable in mfill_atomic_install_pte(), as mk_pte() could return a writable pte (e.g., when VM_SHARED on a shmem file).
--- include/linux/shmem_fs.h | 4 ++-- mm/shmem.c | 4 ++-- mm/userfaultfd.c | 23 ++++++++++++++++++----- 3 files changed, 22 insertions(+), 9 deletions(-)
include/linux/shmem_fs.h +2 -2
@@ -145,11 +145,11 @@ extern int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - bool zeropage, + bool zeropage, bool wp_copy, struct page **pagep);#else /* !CONFIG_SHMEM */ #define shmem_mfill_atomic_pte (dst_mm, dst_pmd, dst_vma, dst_addr, \ - src_addr, zeropage, pagep) ({ BUG(); 0; }) + src_addr, zeropage, wp_copy, pagep) ({ BUG(); 0; })#endif /* CONFIG_SHMEM */ #endif /* CONFIG_USERFAULTFD */
mm/shmem.c +2 -2
@@ -2317,7 +2317,7 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm,struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - bool zeropage, + bool zeropage, bool wp_copy, struct page **pagep) { struct inode *inode = file_inode(dst_vma->vm_file);
@@ -2390,7 +2390,7 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, goto out_release; ret =mfill_atomic_install_pte (dst_mm, dst_pmd, dst_vma, dst_addr, - page, true, false); + page, true, wp_copy);if (ret) goto out_delete_from_cache;
mm/userfaultfd.c +18 -5
@@ -77,10 +77,19 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,* Always mark a PTE as write-protected when needed, regardless of * VM_WRITE, which the user might change. */ -if (wp_copy) +if (wp_copy) { _dst_pte =pte_mkuffd_wp (_dst_pte); -else if (writable) + writable = false; + } + +if (writable) _dst_pte =pte_mkwrite (_dst_pte); +else +/* +* We need this to make sure write bit removed; as mk_pte() +* could return a pte with write bit set. +*/ + _dst_pte =pte_wrprotect (_dst_pte); dst_pte =pte_offset_map_lock (dst_mm, dst_pmd, dst_addr, &ptl);
@@ -95,7 +104,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, } ret = -EEXIST ; -if (!pte_none (*dst_pte)) +/* +* We allow to overwrite a pte marker: consider when both MISSING|WP +* registered, we firstly wr-protect a none pte which has no page cache +* page backing it, then access the page. +*/ +if (!pte_none_mostly (*dst_pte)) goto out_unlock;if (page_in_cache) {
@@ -479,11 +493,10 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, err =mfill_zeropage_pte (dst_mm, dst_pmd, dst_vma, dst_addr); }else { -VM_WARN_ON_ONCE (wp_copy); err =shmem_mfill_atomic_pte (dst_mm, dst_pmd, dst_vma, dst_addr, src_addr, mode !=MCOPY_ATOMIC_NORMAL , - page); + wp_copy, page); }return err;
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 6/23 mm/shmem: Handle uffd-wp special pte in page fault handler
In reply to Peter XuFile-backed memories are prone to unmap/swap so the ptes are always unstable, because they can be easily faulted back later using the page cache. This could lead to uffd-wp getting lost when unmapping or swapping out such memory. One example is shmem. PTE markers are needed to store those information.
This patch prepares it by handling uffd-wp pte markers first it is applied elsewhere, so that the page fault handler can recognize uffd-wp pte markers.
The handling of uffd-wp pte markers is similar to missing fault, it's just that we'll handle this "missing fault" when we see the pte markers, meanwhile we need to make sure the marker information is kept during processing the fault.
This is a slow path of uffd-wp handling, because zapping of wr-protected shmem ptes should be rare. So far it should only trigger in two conditions:
(1) When trying to punch holes in shmem_fallocate(), there is an optimization to zap the pgtables before evicting the page.
(2) When swapping out shmem pages.
Because of this, the page fault handling is simplifed too by not sending the wr-protect message in the 1st page fault, instead the page will be installed read-only, so the uffd-wp message will be generated in the next fault, which will trigger the do_wp_page() path of general uffd-wp handling.
Disable fault-around for all uffd-wp registered ranges for extra safety just like uffd-minor fault, and clean the code up.
--- include/linux/userfaultfd_k.h | 17 +++++++++ mm/memory.c | 67 ++++++++++++++++++++++++++++++----- 2 files changed, 75 insertions(+), 9 deletions(-)
include/linux/userfaultfd_k.h +17 -0
@@ -96,6 +96,18 @@ static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma)return vma->vm_flags & (VM_UFFD_WP |VM_UFFD_MINOR ); } +/* +* Don't do fault around for either WP or MINOR registered uffd range. For +* MINOR registered range, fault around will be a total disaster and ptes can +* be installed without notifications; for WP it should mostly be fine as long +* as the fault around checks for pte_none() before the installation, however +* to be super safe we just forbid it. +*/ +static inline bool uffd_disable_fault_around (struct vm_area_struct *vma) +{ +return vma->vm_flags & (VM_UFFD_WP |VM_UFFD_MINOR ); +} +static inline bool userfaultfd_missing (struct vm_area_struct *vma) {return vma->vm_flags &VM_UFFD_MISSING ;
@@ -236,6 +248,11 @@ static inline void userfaultfd_unmap_complete(struct mm_struct *mm, { } +static inline bool uffd_disable_fault_around (struct vm_area_struct *vma) +{ +return false; +} +#endif /* CONFIG_USERFAULTFD */ static inline bool pte_marker_entry_uffd_wp (swp_entry_t entry)
mm/memory.c +58 -9
105 lines not on this page · raw message
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 7/23 mm/shmem: Persist uffd-wp bit across zapping for file-backed
In reply to Peter XuFile-backed memory is prone to being unmapped at any time. It means all information in the pte will be dropped, including the uffd-wp flag.
To persist the uffd-wp flag, we'll use the pte markers. This patch teaches the zap code to understand uffd-wp and know when to keep or drop the uffd-wp bit.
Add a new flag ZAP_FLAG_DROP_MARKER and set it in zap_details when we don't want to persist such an information, for example, when destroying the whole vma, or punching a hole in a shmem file. For the rest cases we should never drop the uffd-wp bit, or the wr-protect information will get lost.
The new ZAP_FLAG_DROP_MARKER needs to be put into mm.h rather than memory.c because it'll be further referenced in hugetlb files later.
--- include/linux/mm.h | 10 ++++++++ include/linux/mm_inline.h | 43 ++++++++++++++++++++++++++++++++++ mm/memory.c | 49 ++++++++++++++++++++++++++++++++++++--- mm/rmap.c | 8 +++++++ 4 files changed, 107 insertions(+), 3 deletions(-)
include/linux/mm.h +10 -0
@@ -3385,4 +3385,14 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned long start, }#endif +typedef unsigned int __bitwise zap_flags_t ; + +/* +* Whether to drop the pte markers, for example, the uffd-wp information for +* file-backed memory. This should only be specified when we will completely +* drop the page in the mm, either by truncation or unmapping of the vma. By +* default, the flag is not set. +*/ +#define ZAP_FLAG_DROP_MARKER ((__force zap_flags_t) BIT(0)) +#endif /* _LINUX_MM_H */
include/linux/mm_inline.h +43 -0
54 lines not on this page · raw message
mm/memory.c +46 -3
108 lines not on this page · raw message
mm/rmap.c +8 -0
@@ -73,6 +73,7 @@#include <linux/page_idle.h> #include <linux/memremap.h> #include <linux/userfaultfd_k.h> +#include <linux/mm_inline.h> #include <asm/tlbflush.h>
@@ -1526,6 +1527,13 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, pteval =ptep_clear_flush (vma, address, pvmw.pte); } +/* +* Now the pte is cleared. If this is uffd-wp armed pte, we +* may want to replace a none pte with a marker pte if it's +* file-backed, so we don't lose the tracking information. +*/ +pte_install_uffd_wp_if_needed (vma, address, pvmw.pte, pteval); +/* Set the dirty flag on the folio now the pte is gone. */ if (pte_dirty (pteval))folio_mark_dirty (folio);
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 8/23 mm/shmem: Allow uffd wr-protect none pte for file-backed mem
In reply to Peter XuFile-backed memory differs from anonymous memory in that even if the pte is missing, the data could still resides either in the file or in page/swap cache. So when wr-protect a pte, we need to consider none ptes too.
We do that by installing the uffd-wp pte markers when necessary. So when there's a future write to the pte, the fault handler will go the special path to first fault-in the page as read-only, then report to userfaultfd server with the wr-protect message.
On the other hand, when unprotecting a page, it's also possible that the pte got unmapped but replaced by the special uffd-wp marker. Then we'll need to be able to recover from a uffd-wp pte marker into a none pte, so that the next access to the page will fault in correctly as usual when accessed the next time.
Special care needs to be taken throughout the change_protection_range() process. Since now we allow user to wr-protect a none pte, we need to be able to pre-populate the page table entries if we see (!anonymous && MM_CP_UFFD_WP) requests, otherwise change_protection_range() will always skip when the pgtable entry does not exist.
For example, the pgtable can be missing for a whole chunk of 2M pmd, but the page cache can exist for the 2M range. When we want to wr-protect one 4K page within the 2M pmd range, we need to pre-populate the pgtable and install the pte marker showing that we want to get a message and block the thread when the page cache of that 4K page is written. Without pre-populating the pmd, change_protection() will simply skip that whole pmd.
Note that this patch only covers the small pages (pte level) but not covering any of the transparent huge pages yet. That will be done later, and this patch will be a preparation for it too.
--- mm/mprotect.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 62 insertions(+), 2 deletions(-)
mm/mprotect.c +62 -2
@@ -30,6 +30,7 @@#include <linux/mm_inline.h> #include <linux/pgtable.h> #include <linux/sched/sysctl.h> +#include <linux/userfaultfd_k.h> #include <asm/cacheflush.h> #include <asm/mmu_context.h> #include <asm/tlbflush.h>
@@ -184,8 +185,16 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, newpte =pte_swp_mksoft_dirty (newpte);if (pte_swp_uffd_wp (oldpte)) newpte =pte_swp_mkuffd_wp (newpte); - }else if (is_pte_marker_entry (entry)) { -/* Skip it, the same as none pte */ + }else if (pte_marker_entry_uffd_wp (entry)) { +/* +* If this is uffd-wp pte marker and we'd like +* to unprotect it, drop it; the next page +* fault will trigger without uffd trapping. +*/ +if (uffd_wp_resolve) { +pte_clear (vma->vm_mm, addr, pte); + pages++; + }continue ; }else { newpte = oldpte;
@@ -200,6 +209,20 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,set_pte_at (vma->vm_mm, addr, pte, newpte); pages++; } + }else { +/* It must be an none page, or what else?.. */ +WARN_ON_ONCE (!pte_none (oldpte)); +if (unlikely (uffd_wp && !vma_is_anonymous (vma))) { +/* +* For file-backed mem, we need to be able to +* wr-protect a none pte, because even if the +* pte is none, the page/swap cache could +* exist. Doing that by install a marker. +*/ +set_pte_at (vma->vm_mm, addr, pte, +make_pte_marker (PTE_MARKER_UFFD_WP )); + pages++; + } } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();
@@ -233,6 +256,39 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd)return 0 ; } +/* Return true if we're uffd wr-protecting file-backed memory, or false */ +static inline bool +uffd_wp_protect_file (struct vm_area_struct *vma,unsigned long cp_flags) +{ +return (cp_flags &MM_CP_UFFD_WP ) && !vma_is_anonymous (vma); +} + +/* +* If wr-protecting the range for file-backed, populate pgtable for the case +* when pgtable is empty but page cache exists. When {pte|pmd|...}_alloc() +* failed it means no memory, we don't have a better option but stop. +*/ +#define change_pmd_prepare (vma, pmd, cp_flags) \ + do { \ + if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ + if (WARN_ON_ONCE(pte_alloc(vma->vm_mm, pmd))) \ + break; \ + } \ + } while (0) +/* +* This is the general pud/p4d/pgd version of change_pmd_prepare(). We need to +* have separate change_pmd_prepare() because pte_alloc() returns 0 on success, +* while {pmd|pud|p4d}_alloc() returns the valid pointer on success. +*/ +#define change_prepare (vma, high, low, addr, cp_flags) \ + do { \ + if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ + low##_t *p = low##_alloc(vma->vm_mm, high, addr); \ + if (WARN_ON_ONCE(p == NULL)) \ + break; \ + } \ + } while (0) +static inline unsigned long change_pmd_range (struct vm_area_struct *vma,pud_t *pud,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)
@@ -251,6 +307,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, next =pmd_addr_end (addr, end); +change_pmd_prepare (vma, pmd, cp_flags);/* * Automatic NUMA balancing walks the tables with mmap_lock * held for read. It's possible a parallel update to occur
@@ -316,6 +373,7 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma, pud =pud_offset (p4d, addr); do { next = pud_addr_end(addr, end); + change_prepare(vma, pud, pmd, addr, cp_flags); if (pud_none_or_clear_bad(pud)) continue; pages += change_pmd_range(vma, pud, addr, next, newprot,
@@ -336,6 +394,7 @@ static inline unsigned long change_p4d_range(struct vm_area_struct *vma, p4d =p4d_offset (pgd, addr); do { next = p4d_addr_end(addr, end); + change_prepare(vma, p4d, pud, addr, cp_flags); if (p4d_none_or_clear_bad(p4d)) continue; pages += change_pud_range(vma, p4d, addr, next, newprot,
@@ -361,6 +420,7 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,inc_tlb_flush_pending (mm); do { next = pgd_addr_end(addr, end); + change_prepare(vma, pgd, p4d, addr, cp_flags); if (pgd_none_or_clear_bad(pgd)) continue; pages += change_p4d_range(vma, pgd, addr, next, newprot,
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 9/23 mm/shmem: Allows file-back mem to be uffd wr-protected on thps
In reply to Peter XuWe don't have "huge" version of pte markers, instead when necessary we split the thp.
However split the thp is not enough, because file-backed thp is handled totally differently comparing to anonymous thps: rather than doing a real split, the thp pmd will simply got cleared in __split_huge_pmd_locked().
That is not enough if e.g. when there is a thp covers range [0, 2M) but we want to wr-protect small page resides in [4K, 8K) range, because after __split_huge_pmd() returns, there will be a none pmd, and change_pmd_range() will just skip it right after the split.
Here we leverage the previously introduced change_pmd_prepare() macro so that we'll populate the pmd with a pgtable page after the pmd split (in which process the pmd will be cleared for cases like shmem). Then change_pte_range() will do all the rest for us by installing the uffd-wp pte marker at any none pte that we'd like to wr-protect.
--- mm/mprotect.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/mprotect.c +8 -1
@@ -329,8 +329,15 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, }if (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) ||pmd_devmap (*pmd)) { -if (next - addr !=HPAGE_PMD_SIZE ) { +if ((next - addr !=HPAGE_PMD_SIZE ) || +uffd_wp_protect_file (vma, cp_flags)) {__split_huge_pmd (vma, pmd, addr, false,NULL ); +/* +* For file-backed, the pmd could have been +* cleared; make sure pmd populated if +* necessary, then fall-through to pte level. +*/ +change_pmd_prepare (vma, pmd, cp_flags); }else {int nr_ptes =change_huge_pmd (vma, pmd, addr, newprot, cp_flags);
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 10/23 mm/shmem: Handle uffd-wp during fork()
In reply to Peter XuNormally we skip copy page when fork() for VM_SHARED shmem, but we can't skip it anymore if uffd-wp is enabled on dst vma. This should only happen when the src uffd has UFFD_FEATURE_EVENT_FORK enabled on uffd-wp shmem vma, so that VM_UFFD_WP will be propagated onto dst vma too, then we should copy the pgtables with uffd-wp bit and pte markers, because these information will be lost otherwise.
Since the condition checks will become even more complicated for deciding "whether a vma needs to copy the pgtable during fork()", introduce a helper vma_needs_copy() for it, so everything will be clearer.
--- mm/memory.c | 49 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 8 deletions(-)
mm/memory.c +41 -8
@@ -857,6 +857,14 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,if (try_restore_exclusive_pte (src_pte, src_vma, addr))return -EBUSY ;return -ENOENT ; + }else if (is_pte_marker_entry (entry)) { +/* +* We're copying the pgtable should only because dst_vma has +* uffd-wp enabled, do sanity check. +*/ +WARN_ON_ONCE (!userfaultfd_wp (dst_vma)); +set_pte_at (dst_mm, addr, dst_pte, pte); +return 0 ; }if (!userfaultfd_wp (dst_vma)) pte =pte_swp_clear_uffd_wp (pte);
@@ -1225,6 +1233,38 @@ copy_p4d_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,return 0 ; } +/* +* Return true if the vma needs to copy the pgtable during this fork(). Return +* false when we can speed up fork() by allowing lazy page faults later until +* when the child accesses the memory range. +*/ +bool +vma_needs_copy (struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma) +{ +/* +* Always copy pgtables when dst_vma has uffd-wp enabled even if it's +* file-backed (e.g. shmem). Because when uffd-wp is enabled, pgtable +* contains uffd-wp protection information, that's something we can't +* retrieve from page cache, and skip copying will lose those info. +*/ +if (userfaultfd_wp (dst_vma)) +return true; + +if (src_vma->vm_flags & (VM_HUGETLB |VM_PFNMAP |VM_MIXEDMAP )) +return true; + +if (src_vma->anon_vma) +return true; + +/* +* Don't copy ptes where a page fault will fill them correctly. Fork +* becomes much lighter when there are big shared or private readonly +* mappings. The tradeoff is that copy_page_range is more efficient +* than faulting. +*/ +return false; +} +int copy_page_range (struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma) {
@@ -1238,14 +1278,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)bool is_cow;int ret; -/* -* Don't copy ptes where a page fault will fill them correctly. -* Fork becomes much lighter when there are big shared or private -* readonly mappings. The tradeoff is that copy_page_range is more -* efficient than faulting. -*/ -if (!(src_vma->vm_flags & (VM_HUGETLB |VM_PFNMAP |VM_MIXEDMAP )) && - !src_vma->anon_vma) +if (!vma_needs_copy (dst_vma, src_vma))return 0 ;if (is_vm_hugetlb_page (src_vma))
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 11/23 mm/hugetlb: Introduce huge pte version of uffd-wp helpers
In reply to Peter XuThey will be used in the follow up patches to either check/set/clear uffd-wp bit of a huge pte.
So far it reuses all the small pte helpers. Archs can overwrite these versions when necessary (with __HAVE_ARCH_HUGE_PTE_UFFD_WP* macros) in the future.
--- arch/s390/include/asm/hugetlb.h | 15 +++++++++++++++ include/asm-generic/hugetlb.h | 15 +++++++++++++++ 2 files changed, 30 insertions(+)
arch/s390/include/asm/hugetlb.h +15 -0
@@ -115,6 +115,21 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot)return pte_modify (pte, newprot); } +static inline pte_t huge_pte_mkuffd_wp (pte_t pte) +{ +return pte; +} + +static inline pte_t huge_pte_clear_uffd_wp (pte_t pte) +{ +return pte; +} + +static inline int huge_pte_uffd_wp (pte_t pte) +{ +return 0 ; +} +static inline bool gigantic_page_runtime_supported (void ) {return true;
include/asm-generic/hugetlb.h +15 -0
@@ -35,6 +35,21 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot)return pte_modify (pte, newprot); } +static inline pte_t huge_pte_mkuffd_wp (pte_t pte) +{ +return pte_mkuffd_wp (pte); +} + +static inline pte_t huge_pte_clear_uffd_wp (pte_t pte) +{ +return pte_clear_uffd_wp (pte); +} + +static inline int huge_pte_uffd_wp (pte_t pte) +{ +return pte_uffd_wp (pte); +} +#ifndef __HAVE_ARCH_HUGE_PTE_CLEARstatic inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz)
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 12/23 mm/hugetlb: Hook page faults for uffd write protection
In reply to Peter XuHook up hugetlbfs_fault() with the capability to handle userfaultfd-wp faults.
We do this slightly earlier than hugetlb_cow() so that we can avoid taking some extra locks that we definitely don't need.
--- mm/hugetlb.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
mm/hugetlb.c +20 -0
@@ -5680,6 +5680,26 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!pte_same (entry,huge_ptep_get (ptep)))) goto out_ptl; +/* Handle userfault-wp first, before trying to lock more pages */ +if (userfaultfd_wp (vma) &&huge_pte_uffd_wp (huge_ptep_get (ptep)) && + (flags &FAULT_FLAG_WRITE ) && !huge_pte_write (entry)) { +struct vm_fault vmf = { + .vma = vma, + .address = haddr, + .real_address = address, + .flags = flags, + }; + +spin_unlock (ptl); +if (pagecache_page) { +unlock_page (pagecache_page); +put_page (pagecache_page); + } +mutex_unlock (&hugetlb_fault_mutex_table[hash]); +i_mmap_unlock_read (mapping); +return handle_userfault (&vmf,VM_UFFD_WP ); + } +/* * hugetlb_cow() requires page locks of pte_page(entry) and * pagecache_page, so here we need take the former one
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 13/23 mm/hugetlb: Take care of UFFDIO_COPY_MODE_WP
In reply to Peter XuPass the wp_copy variable into hugetlb_mcopy_atomic_pte() thoughout the stack. Apply the UFFD_WP bit if UFFDIO_COPY_MODE_WP is with UFFDIO_COPY.
Hugetlb pages are only managed by hugetlbfs, so we're safe even without setting dirty bit in the huge pte if the page is installed as read-only. However we'd better still keep the dirty bit set for a read-only UFFDIO_COPY pte (when UFFDIO_COPY_MODE_WP bit is set), not only to match what we do with shmem, but also because the page does contain dirty data that the kernel just copied from the userspace.
--- include/linux/hugetlb.h | 6 ++++-- mm/hugetlb.c | 29 +++++++++++++++++++++++------ mm/userfaultfd.c | 14 +++++++++----- 3 files changed, 36 insertions(+), 13 deletions(-)
include/linux/hugetlb.h +4 -2
@@ -160,7 +160,8 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte,unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep); + struct page **pagep, + bool wp_copy);#endif /* CONFIG_USERFAULTFD */ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, struct vm_area_struct *vma,
@@ -355,7 +356,8 @@ static inline int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep) + struct page **pagep, + bool wp_copy) {BUG ();return 0 ;
mm/hugetlb.c +23 -6
@@ -5763,7 +5763,8 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep) + struct page **pagep, + bool wp_copy) { bool is_continue = (mode == MCOPY_ATOMIC_CONTINUE);struct hstate *h =hstate_vma (dst_vma);
@@ -5893,7 +5894,12 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, goto out_release_unlock; ret = -EEXIST ; -if (!huge_pte_none (huge_ptep_get (dst_pte))) +/* +* We allow to overwrite a pte marker: consider when both MISSING|WP +* registered, we firstly wr-protect a none pte which has no page cache +* page backing it, then access the page. +*/ +if (!huge_pte_none_mostly (huge_ptep_get (dst_pte))) goto out_release_unlock;if (vm_shared) {
@@ -5903,17 +5909,28 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,hugepage_add_new_anon_rmap (page, dst_vma, dst_addr); } -/* For CONTINUE on a non-shared VMA, don't set VM_WRITE for CoW. */ -if (is_continue && !vm_shared) +/* +* For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY +* with wp flag set, don't set pte write bit. +*/ +if (wp_copy || (is_continue && !vm_shared)) writable =0 ;else writable = dst_vma->vm_flags &VM_WRITE ; _dst_pte =make_huge_pte (dst_vma, page, writable); -if (writable) - _dst_pte =huge_pte_mkdirty (_dst_pte); +/* +* Always mark UFFDIO_COPY page dirty; note that this may not be +* extremely important for hugetlbfs for now since swapping is not +* supported, but we should still be clear in that this page cannot be +* thrown away at will, even if write bit not set. +*/ + _dst_pte =huge_pte_mkdirty (_dst_pte); _dst_pte =pte_mkyoung (_dst_pte); +if (wp_copy) + _dst_pte =huge_pte_mkuffd_wp (_dst_pte); +set_huge_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte); (void)huge_ptep_set_access_flags(dst_vma, dst_addr, dst_pte, _dst_pte,
mm/userfaultfd.c +9 -5
@@ -304,7 +304,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,unsigned long dst_start, unsigned long src_start, unsigned long len, - enum mcopy_atomic_mode mode) + enum mcopy_atomic_mode mode, + bool wp_copy) { int vm_shared = dst_vma->vm_flags & VM_SHARED;ssize_t err;
@@ -392,7 +393,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, }if (mode !=MCOPY_ATOMIC_CONTINUE && - !huge_pte_none (huge_ptep_get (dst_pte))) { + !huge_pte_none_mostly (huge_ptep_get (dst_pte))) { err = -EEXIST ;mutex_unlock (&hugetlb_fault_mutex_table[hash]);i_mmap_unlock_read (mapping);
@@ -400,7 +401,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, } err =hugetlb_mcopy_atomic_pte (dst_mm, dst_pte, dst_vma, - dst_addr, src_addr, mode, &page); + dst_addr, src_addr, mode, &page, + wp_copy);mutex_unlock (&hugetlb_fault_mutex_table[hash]);i_mmap_unlock_read (mapping);
@@ -455,7 +457,8 @@ extern ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,unsigned long dst_start, unsigned long src_start, unsigned long len, - enum mcopy_atomic_mode mode); + enum mcopy_atomic_mode mode, + bool wp_copy);#endif /* CONFIG_HUGETLB_PAGE */ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm,
@@ -575,7 +578,8 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,*/ if (is_vm_hugetlb_page (dst_vma))return __mcopy_atomic_hugetlb (dst_mm, dst_vma, dst_start, - src_start, len, mcopy_mode); + src_start, len, mcopy_mode, + wp_copy);if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) goto out_unlock;
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 14/23 mm/hugetlb: Handle UFFDIO_WRITEPROTECT
In reply to Peter XuThis starts from passing cp_flags into hugetlb_change_protection() so hugetlb will be able to handle MM_CP_UFFD_WP[_RESOLVE] requests.
huge_pte_clear_uffd_wp() is introduced to handle the case where the UFFDIO_WRITEPROTECT is requested upon migrating huge page entries.
--- include/linux/hugetlb.h | 6 ++++-- mm/hugetlb.c | 13 ++++++++++++- mm/mprotect.c | 3 ++- mm/userfaultfd.c | 8 ++++++++ 4 files changed, 26 insertions(+), 4 deletions(-)
include/linux/hugetlb.h +4 -2
@@ -210,7 +210,8 @@ struct page *follow_huge_pgd(struct mm_struct *mm, unsigned long address,int pmd_huge (pmd_t pmd);int pud_huge (pud_t pud);unsigned long hugetlb_change_protection (struct vm_area_struct *vma, -unsigned long address,unsigned long end,pgprot_t newprot); +unsigned long address,unsigned long end,pgprot_t newprot, +unsigned long cp_flags);bool is_hugetlb_entry_migration (pte_t pte);void hugetlb_unshare_all_pmds (struct vm_area_struct *vma);
@@ -391,7 +392,8 @@ static inline void move_hugetlb_state(struct page *oldpage,static inline unsigned long hugetlb_change_protection (struct vm_area_struct *vma,unsigned long address, -unsigned long end,pgprot_t newprot) +unsigned long end,pgprot_t newprot, +unsigned long cp_flags) {return 0 ; }
mm/hugetlb.c +12 -1
@@ -6151,7 +6151,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, }unsigned long hugetlb_change_protection (struct vm_area_struct *vma, -unsigned long address,unsigned long end,pgprot_t newprot) +unsigned long address,unsigned long end, +pgprot_t newprot,unsigned long cp_flags) {struct mm_struct *mm = vma->vm_mm;unsigned long start = address;
@@ -6161,6 +6162,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,unsigned long pages =0 ;bool shared_pmd = false;struct mmu_notifier_range range; +bool uffd_wp = cp_flags &MM_CP_UFFD_WP ; +bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ;/* * In the case of shared PMDs, the area to flush could be beyond
@@ -6202,6 +6205,10 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, entry =make_readable_migration_entry (swp_offset (entry)); newpte =swp_entry_to_pte (entry); +if (uffd_wp) + newpte =pte_swp_mkuffd_wp (newpte); +else if (uffd_wp_resolve) + newpte =pte_swp_clear_uffd_wp (newpte);set_huge_swap_pte_at (mm, address, ptep, newpte,huge_page_size (h)); pages++;
@@ -6216,6 +6223,10 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, old_pte =huge_ptep_modify_prot_start (vma, address, ptep); pte =huge_pte_modify (old_pte, newprot); pte =arch_make_huge_pte (pte, shift, vma->vm_flags); +if (uffd_wp) + pte =huge_pte_mkuffd_wp (huge_pte_wrprotect (pte)); +else if (uffd_wp_resolve) + pte =huge_pte_clear_uffd_wp (pte);huge_ptep_modify_prot_commit (vma, address, ptep, old_pte, pte); pages++; }
mm/mprotect.c +2 -1
@@ -451,7 +451,8 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start,BUG_ON ((cp_flags &MM_CP_UFFD_WP_ALL ) ==MM_CP_UFFD_WP_ALL );if (is_vm_hugetlb_page (vma)) - pages =hugetlb_change_protection (vma, start, end, newprot); + pages =hugetlb_change_protection (vma, start, end, newprot, + cp_flags);else pages =change_protection_range (vma, start, end, newprot, cp_flags);
mm/userfaultfd.c +8 -0
@@ -704,6 +704,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, atomic_t *mmap_changing) {struct vm_area_struct *dst_vma; +unsigned long page_mask;pgprot_t newprot;int err;
@@ -740,6 +741,13 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,if (!vma_is_anonymous (dst_vma)) goto out_unlock; +if (is_vm_hugetlb_page (dst_vma)) { + err = -EINVAL ; + page_mask =vma_kernel_pagesize (dst_vma) -1 ; +if ((start & page_mask) || (len & page_mask)) + goto out_unlock; + } +if (enable_wp) newprot =vm_get_page_prot (dst_vma->vm_flags & ~(VM_WRITE ));else
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 15/23 mm/hugetlb: Handle pte markers in page faults
In reply to Peter XuAllow hugetlb code to handle pte markers just like none ptes. It's mostly there, we just need to make sure we don't assume hugetlb_no_page() only handles none pte, so when detecting pte change we should use pte_same() rather than pte_none(). We need to pass in the old_pte to do the comparison.
Check the original pte to see whether it's a pte marker, if it is, we should recover uffd-wp bit on the new pte to be installed, so that the next write will be trapped by uffd.
--- mm/hugetlb.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
mm/hugetlb.c +14 -4
@@ -5383,7 +5383,8 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma,static vm_fault_t hugetlb_no_page (struct mm_struct *mm,struct vm_area_struct *vma,struct address_space *mapping,pgoff_t idx, -unsigned long address,pte_t *ptep,unsigned int flags) +unsigned long address,pte_t *ptep, +pte_t old_pte,unsigned int flags) {struct hstate *h =hstate_vma (vma);vm_fault_t ret =VM_FAULT_SIGBUS ;
@@ -5509,7 +5510,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, ptl =huge_pte_lock (h, mm, ptep); ret =0 ; -if (!huge_pte_none (huge_ptep_get (ptep))) +/* If pte changed from under us, retry */ +if (!pte_same (huge_ptep_get (ptep), old_pte)) goto backout;if (anon_rmap) {
@@ -5519,6 +5521,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,page_dup_rmap (page, true); new_pte =make_huge_pte (vma, page, ((vma->vm_flags &VM_WRITE ) && (vma->vm_flags &VM_SHARED ))); +/* +* If this pte was previously wr-protected, keep it wr-protected even +* if populated. +*/ +if (unlikely (pte_marker_uffd_wp (old_pte))) + new_pte =huge_pte_wrprotect (huge_pte_mkuffd_wp (new_pte));set_huge_pte_at (mm, haddr, ptep, new_pte);hugetlb_count_add (pages_per_huge_page (h), mm);
@@ -5636,8 +5644,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,mutex_lock (&hugetlb_fault_mutex_table[hash]); entry =huge_ptep_get (ptep); -if (huge_pte_none (entry)) { - ret =hugetlb_no_page (mm, vma, mapping, idx, address, ptep, flags); +/* PTE markers should be handled the same way as none pte */ +if (huge_pte_none_mostly (entry)) { + ret =hugetlb_no_page (mm, vma, mapping, idx, address, ptep, + entry, flags); goto out_mutex; }
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 16/23 mm/hugetlb: Allow uffd wr-protect none ptes
In reply to Peter XuTeach hugetlbfs code to wr-protect none ptes just in case the page cache existed for that pte. Meanwhile we also need to be able to recognize a uffd-wp marker pte and remove it for uffd_wp_resolve.
Since at it, introduce a variable "psize" to replace all references to the huge page size fetcher.
--- mm/hugetlb.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
mm/hugetlb.c +24 -4
@@ -6169,7 +6169,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,pte_t *ptep;pte_t pte;struct hstate *h =hstate_vma (vma); -unsigned long pages =0 ; +unsigned long pages =0 , psize =huge_page_size (h);bool shared_pmd = false;struct mmu_notifier_range range;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;
@@ -6189,13 +6189,19 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,mmu_notifier_invalidate_range_start (&range);i_mmap_lock_write (vma->vm_file->f_mapping); -for (; address < end; address +=huge_page_size (h)) { +for (; address < end; address += psize) {spinlock_t *ptl; - ptep =huge_pte_offset (mm, address,huge_page_size (h)); + ptep =huge_pte_offset (mm, address, psize);if (!ptep)continue ; ptl =huge_pte_lock (h, mm, ptep);if (huge_pmd_unshare (mm, vma, &address, ptep)) { +/* +* When uffd-wp is enabled on the vma, unshare +* shouldn't happen at all. Warn about it if it +* happened due to some reason. +*/ +WARN_ON_ONCE (uffd_wp || uffd_wp_resolve); pages++;spin_unlock (ptl); shared_pmd = true;
@@ -6220,12 +6226,20 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,else if (uffd_wp_resolve) newpte =pte_swp_clear_uffd_wp (newpte);set_huge_swap_pte_at (mm, address, ptep, - newpte,huge_page_size (h)); + newpte, psize); pages++; }spin_unlock (ptl);continue ; } +if (unlikely (pte_marker_uffd_wp (pte))) { +/* +* This is changing a non-present pte into a none pte, +* no need for huge_ptep_modify_prot_start/commit(). +*/ +if (uffd_wp_resolve) +huge_pte_clear (mm, address, ptep, psize); + }if (!huge_pte_none (pte)) {pte_t old_pte;unsigned int shift =huge_page_shift (hstate_vma (vma));
@@ -6239,6 +6253,12 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pte =huge_pte_clear_uffd_wp (pte);huge_ptep_modify_prot_commit (vma, address, ptep, old_pte, pte); pages++; + }else { +/* None pte */ +if (unlikely (uffd_wp)) +/* Safe to modify directly (none->non-present). */ +set_huge_pte_at (mm, address, ptep, +make_pte_marker (PTE_MARKER_UFFD_WP )); }spin_unlock (ptl); }
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 17/23 mm/hugetlb: Only drop uffd-wp special pte if required
In reply to Peter XuAs with shmem uffd-wp special ptes, only drop the uffd-wp special swap pte if unmapping an entire vma or synchronized such that faults can not race with the unmap operation. This requires passing zap_flags all the way to the lowest level hugetlb unmap routine: __unmap_hugepage_range.
In general, unmap calls originated in hugetlbfs code will pass the ZAP_FLAG_DROP_MARKER flag as synchronization is in place to prevent faults. The exception is hole punch which will first unmap without any synchronization. Later when hole punch actually removes the page from the file, it will check to see if there was a subsequent fault and if so take the hugetlb fault mutex while unmapping again. This second unmap will pass in ZAP_FLAG_DROP_MARKER.
The justification of "whether to apply ZAP_FLAG_DROP_MARKER flag when unmap a hugetlb range" is (IMHO): we should never reach a state when a page fault could errornously fault in a page-cache page that was wr-protected to be writable, even in an extremely short period. That could happen if e.g. we pass ZAP_FLAG_DROP_MARKER when hugetlbfs_punch_hole() calls hugetlb_vmdelete_list(), because if a page faults after that call and before remove_inode_hugepages() is executed, the page cache can be mapped writable again in the small racy window, that can cause unexpected data overwritten.
--- fs/hugetlbfs/inode.c | 15 +++++++++------ include/linux/hugetlb.h | 8 +++++--- mm/hugetlb.c | 33 +++++++++++++++++++++++++-------- mm/memory.c | 5 ++++- 4 files changed, 43 insertions(+), 18 deletions(-)
fs/hugetlbfs/inode.c +9 -6
@@ -404,7 +404,8 @@ static void remove_huge_page(struct page *page) }static void -hugetlb_vmdelete_list (struct rb_root_cached *root,pgoff_t start,pgoff_t end) +hugetlb_vmdelete_list (struct rb_root_cached *root,pgoff_t start,pgoff_t end, +unsigned long zap_flags) {struct vm_area_struct *vma;
@@ -438,7 +439,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) }unmap_hugepage_range (vma, vma->vm_start + v_offset, v_end, -NULL ); +NULL , zap_flags); } }
@@ -516,7 +517,8 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,mutex_lock (&hugetlb_fault_mutex_table[hash]);hugetlb_vmdelete_list (&mapping->i_mmap, index *pages_per_huge_page (h), - (index +1 ) *pages_per_huge_page (h)); + (index +1 ) *pages_per_huge_page (h), +ZAP_FLAG_DROP_MARKER );i_mmap_unlock_write (mapping); }
@@ -582,7 +584,8 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset)i_mmap_lock_write (mapping);i_size_write (inode, offset);if (!RB_EMPTY_ROOT (&mapping->i_mmap.rb_root)) -hugetlb_vmdelete_list (&mapping->i_mmap, pgoff,0 ); +hugetlb_vmdelete_list (&mapping->i_mmap, pgoff,0 , +ZAP_FLAG_DROP_MARKER );i_mmap_unlock_write (mapping);remove_inode_hugepages (inode, offset,LLONG_MAX ); }
@@ -615,8 +618,8 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len)i_mmap_lock_write (mapping);if (!RB_EMPTY_ROOT (&mapping->i_mmap.rb_root))hugetlb_vmdelete_list (&mapping->i_mmap, - hole_start >>PAGE_SHIFT , - hole_end >>PAGE_SHIFT ); + hole_start >>PAGE_SHIFT , + hole_end >>PAGE_SHIFT ,0 );i_mmap_unlock_write (mapping);remove_inode_hugepages (inode, hole_start, hole_end);inode_unlock (inode);
include/linux/hugetlb.h +5 -3
@@ -143,11 +143,12 @@ long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, unsigned long *, unsigned long *, long, unsigned int, int *);void unmap_hugepage_range (struct vm_area_struct *, -unsigned long ,unsigned long ,struct page *); +unsigned long ,unsigned long ,struct page *, +unsigned long );void __unmap_hugepage_range_final (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long start,unsigned long end, -struct page *ref_page); +struct page *ref_page,unsigned long zap_flags);void hugetlb_report_meminfo (struct seq_file *);int hugetlb_report_node_meminfo (char *buf,int len,int nid);void hugetlb_show_meminfo (void );
@@ -400,7 +401,8 @@ static inline unsigned long hugetlb_change_protection(static inline void __unmap_hugepage_range_final (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long start, -unsigned long end,struct page *ref_page) +unsigned long end,struct page *ref_page, +unsigned long zap_flags) {BUG (); }
mm/hugetlb.c +25 -8
@@ -4931,7 +4931,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,static void __unmap_hugepage_range (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long start,unsigned long end, -struct page *ref_page) +struct page *ref_page,unsigned long zap_flags) {struct mm_struct *mm = vma->vm_mm;unsigned long address;
@@ -4987,7 +4987,18 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct* unmapped and its refcount is dropped, so just clear pte here. */ if (unlikely (!pte_present (pte))) { -huge_pte_clear (mm, address, ptep, sz); +/* +* If the pte was wr-protected by uffd-wp in any of the +* swap forms, meanwhile the caller does not want to +* drop the uffd-wp bit in this zap, then replace the +* pte with a marker. +*/ +if (pte_swp_uffd_wp_any (pte) && + !(zap_flags &ZAP_FLAG_DROP_MARKER )) +set_huge_pte_at (mm, address, ptep, +make_pte_marker (PTE_MARKER_UFFD_WP )); +else +huge_pte_clear (mm, address, ptep, sz);spin_unlock (ptl);continue ; }
@@ -5015,7 +5026,11 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_structtlb_remove_huge_tlb_entry (h, tlb, ptep, address);if (huge_pte_dirty (pte))set_page_dirty (page); - +/* Leave a uffd-wp pte marker if needed */ +if (huge_pte_uffd_wp (pte) && + !(zap_flags &ZAP_FLAG_DROP_MARKER )) +set_huge_pte_at (mm, address, ptep, +make_pte_marker (PTE_MARKER_UFFD_WP ));hugetlb_count_sub (pages_per_huge_page (h), mm);page_remove_rmap (page, vma, true);
@@ -5049,9 +5064,10 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_structvoid __unmap_hugepage_range_final (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long start, -unsigned long end,struct page *ref_page) +unsigned long end,struct page *ref_page, +unsigned long zap_flags) { -__unmap_hugepage_range (tlb, vma, start, end, ref_page); +__unmap_hugepage_range (tlb, vma, start, end, ref_page, zap_flags);/* * Clear this flag so that x86's huge_pmd_share page_table_shareable
@@ -5067,12 +5083,13 @@ void __unmap_hugepage_range_final(struct mmu_gather *tlb, }void unmap_hugepage_range (struct vm_area_struct *vma,unsigned long start, -unsigned long end,struct page *ref_page) +unsigned long end,struct page *ref_page, +unsigned long zap_flags) {struct mmu_gather tlb;tlb_gather_mmu (&tlb, vma->vm_mm); -__unmap_hugepage_range (&tlb, vma, start, end, ref_page); +__unmap_hugepage_range (&tlb, vma, start, end, ref_page, zap_flags);tlb_finish_mmu (&tlb); }
@@ -5127,7 +5144,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma,*/ if (!is_vma_resv_set (iter_vma,HPAGE_RESV_OWNER ))unmap_hugepage_range (iter_vma, address, - address +huge_page_size (h), page); + address +huge_page_size (h), page,0 ); }i_mmap_unlock_write (mapping); }
mm/memory.c +4 -1
@@ -1679,8 +1679,11 @@ static void unmap_single_vma(struct mmu_gather *tlb,* safe to do nothing in this case. */ if (vma->vm_file) { +unsigned long zap_flags = details ? + details->zap_flags :0 ;i_mmap_lock_write (vma->vm_file->f_mapping); -__unmap_hugepage_range_final (tlb, vma, start, end,NULL ); +__unmap_hugepage_range_final (tlb, vma, start, end, +NULL , zap_flags);i_mmap_unlock_write (vma->vm_file->f_mapping); } }else
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 18/23 mm/hugetlb: Handle uffd-wp during fork()
In reply to Peter XuFirstly, we'll need to pass in dst_vma into copy_hugetlb_page_range() because for uffd-wp it's the dst vma that matters on deciding how we should treat uffd-wp protected ptes.
We should recognize pte markers during fork and do the pte copy if needed.
--- include/linux/hugetlb.h | 7 +++++-- mm/hugetlb.c | 41 +++++++++++++++++++++++++++-------------- mm/memory.c | 2 +- 3 files changed, 33 insertions(+), 17 deletions(-)
include/linux/hugetlb.h +5 -2
@@ -137,7 +137,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,struct vm_area_struct *new_vma, unsigned long old_addr, unsigned long new_addr, unsigned long len); -int copy_hugetlb_page_range (struct mm_struct *,struct mm_struct *,struct vm_area_struct *); +int copy_hugetlb_page_range (struct mm_struct *,struct mm_struct *, +struct vm_area_struct *,struct vm_area_struct *); long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, struct vm_area_struct **, unsigned long *, unsigned long *, long, unsigned int,
@@ -268,7 +269,9 @@ static inline struct page *follow_huge_addr(struct mm_struct *mm, }static inline int copy_hugetlb_page_range (struct mm_struct *dst, -struct mm_struct *src,struct vm_area_struct *vma) +struct mm_struct *src, +struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma) {BUG ();return 0 ;
mm/hugetlb.c +27 -14
111 lines not on this page · raw message
mm/memory.c +1 -1
@@ -1282,7 +1282,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)return 0 ;if (is_vm_hugetlb_page (src_vma)) -return copy_hugetlb_page_range (dst_mm, src_mm, src_vma); +return copy_hugetlb_page_range (dst_mm, src_mm, dst_vma, src_vma);if (unlikely (src_vma->vm_flags &VM_PFNMAP )) {/*
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 19/23 mm/khugepaged: Don't recycle vma pgtable if uffd-wp registered
In reply to Peter XuWhen we're trying to collapse a 2M huge shmem page, don't retract pgtable pmd page if it's registered with uffd-wp, because that pgtable could have pte markers installed. Recycling of that pgtable means we'll lose the pte markers. That could cause data loss for an uffd-wp enabled application on shmem.
Instead of disabling khugepaged on these files, simply skip retracting these special VMAs, then the page cache can still be merged into a huge thp, and other mm/vma can still map the range of file with a huge thp when proper.
Note that checking VM_UFFD_WP needs to be done with mmap_sem held for write, that avoids race like:
khugepaged user thread
========== ===========
check VM_UFFD_WP, not set
UFFDIO_REGISTER with uffd-wp on shmem wr-protect some pages (install markers)
take mmap_sem write lock
erase pmd and free pmd page
--> pte markers are dropped unnoticed!
--- mm/khugepaged.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
mm/khugepaged.c +13 -1
@@ -1456,6 +1456,10 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr)if (!hugepage_vma_check (vma, vma->vm_flags |VM_HUGEPAGE ))return ; +/* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ +if (userfaultfd_wp (vma)) +return ; + hpage =find_lock_page (vma->vm_file->f_mapping,linear_page_index (vma, haddr));if (!hpage)
@@ -1591,7 +1595,15 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)* reverse order. Trylock is a way to avoid deadlock. */ if (mmap_write_trylock (mm)) { -if (!khugepaged_test_exit (mm)) +/* +* When a vma is registered with uffd-wp, we can't +* recycle the pmd pgtable because there can be pte +* markers installed. Skip it only, so the rest mm/vma +* can still have the same file mapped hugely, however +* it'll always mapped in small page size for uffd-wp +* registered ranges. +*/ +if (!khugepaged_test_exit (mm) && !userfaultfd_wp (vma))collapse_and_free_pmd (mm, vma, addr, pmd);mmap_write_unlock (mm); }else {
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 20/23 mm/pagemap: Recognize uffd-wp bit for shmem/hugetlbfs
In reply to Peter XuThis requires the pagemap code to be able to recognize the newly introduced swap special pte for uffd-wp, meanwhile the general case for hugetlb that we recently start to support. It should make pagemap uffd-wp support complete.
--- fs/proc/task_mmu.c | 7 +++++++ 1 file changed, 7 insertions(+)
fs/proc/task_mmu.c +7 -0
@@ -1421,6 +1421,8 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, migration =is_migration_entry (entry);if (is_pfn_swap_entry (entry)) page =pfn_swap_entry_to_page (entry); +if (pte_marker_entry_uffd_wp (entry)) + flags |=PM_UFFD_WP ; }if (page && !PageAnon (page))
@@ -1556,10 +1558,15 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,if (page_mapcount (page) ==1 ) flags |=PM_MMAP_EXCLUSIVE ; +if (huge_pte_uffd_wp (pte)) + flags |=PM_UFFD_WP ; + flags |=PM_PRESENT ;if (pm->show_pfn) frame =pte_pfn (pte) + ((addr & ~hmask) >>PAGE_SHIFT ); + }else if (pte_swp_uffd_wp_any (pte)) { + flags |=PM_UFFD_WP ; }for (; addr != end; addr +=PAGE_SIZE ) {
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 21/23 mm/uffd: Enable write protection for shmem & hugetlbfs
In reply to Peter XuWe've had all the necessary changes ready for both shmem and hugetlbfs. Turn on all the shmem/hugetlbfs switches for userfaultfd-wp.
We can expand UFFD_API_RANGE_IOCTLS_BASIC with _UFFDIO_WRITEPROTECT too because all existing types now support write protection mode.
Since vma_can_userfault() will be used elsewhere, move into userfaultfd_k.h.
--- fs/userfaultfd.c | 21 ++------------------- include/linux/userfaultfd_k.h | 11 +++++++++++ include/uapi/linux/userfaultfd.h | 10 ++++++++-- mm/userfaultfd.c | 9 +++------ 4 files changed, 24 insertions(+), 27 deletions(-)
fs/userfaultfd.c +2 -19
@@ -1257,24 +1257,6 @@ static __always_inline int validate_range(struct mm_struct *mm,return 0 ; } -static inline bool vma_can_userfault (struct vm_area_struct *vma, -unsigned long vm_flags) -{ -/* FIXME: add WP support to hugetlbfs and shmem */ -if (vm_flags &VM_UFFD_WP ) { -if (is_vm_hugetlb_page (vma) ||vma_is_shmem (vma)) -return false; - } - -if (vm_flags &VM_UFFD_MINOR ) { -if (!(is_vm_hugetlb_page (vma) ||vma_is_shmem (vma))) -return false; - } - -return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || -vma_is_shmem (vma); -} -static int userfaultfd_register (struct userfaultfd_ctx *ctx,unsigned long arg) {
@@ -1954,7 +1936,8 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM );#endif #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP - uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP ; + uffdio_api.features &= + ~(UFFD_FEATURE_PAGEFAULT_FLAG_WP |UFFD_FEATURE_WP_HUGETLBFS_SHMEM );#endif uffdio_api.ioctls =UFFD_API_IOCTLS ; ret = -EFAULT ;
include/linux/userfaultfd_k.h +11 -0
@@ -18,6 +18,7 @@#include <linux/swap.h> #include <linux/swapops.h> #include <asm-generic/pgtable_uffd.h> +#include <linux/hugetlb_inline.h> /* The set of all possible UFFD-related VM flags. */ #define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)
@@ -140,6 +141,16 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return vma->vm_flags & __VM_UFFD_FLAGS; } +static inline bool vma_can_userfault (struct vm_area_struct *vma, +unsigned long vm_flags) +{ +if (vm_flags &VM_UFFD_MINOR ) +return is_vm_hugetlb_page (vma) ||vma_is_shmem (vma); + +return vma_is_anonymous (vma) ||is_vm_hugetlb_page (vma) || +vma_is_shmem (vma); +} +extern int dup_userfaultfd (struct vm_area_struct *,struct list_head *);extern void dup_userfaultfd_complete (struct list_head *);
include/uapi/linux/userfaultfd.h +8 -2
@@ -33,7 +33,8 @@ UFFD_FEATURE_THREAD_ID | \ UFFD_FEATURE_MINOR_HUGETLBFS | \ UFFD_FEATURE_MINOR_SHMEM | \ - UFFD_FEATURE_EXACT_ADDRESS) + UFFD_FEATURE_EXACT_ADDRESS | \ + UFFD_FEATURE_WP_HUGETLBFS_SHMEM)#define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \
@@ -47,7 +48,8 @@#define UFFD_API_RANGE_IOCTLS_BASIC \ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY | \ - (__u64)1 << _UFFDIO_CONTINUE) + (__u64)1 << _UFFDIO_CONTINUE | \ + (__u64)1 << _UFFDIO_WRITEPROTECT)/* * Valid ioctl command number range with this API is from 0x00 to
@@ -194,6 +196,9 @@ struct uffdio_api {* UFFD_FEATURE_EXACT_ADDRESS indicates that the exact address of page * faults would be provided and the offset within the page would not be * masked. +* +* UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd +* write-protection mode is supported on both shmem and hugetlbfs. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)#define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -207,6 +212,7 @@ struct uffdio_api {#define UFFD_FEATURE_MINOR_HUGETLBFS (1<<9)#define UFFD_FEATURE_MINOR_SHMEM (1<<10)#define UFFD_FEATURE_EXACT_ADDRESS (1<<11) +#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)__u64 features;__u64 ioctls;
mm/userfaultfd.c +3 -6
@@ -730,15 +730,12 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, err = -ENOENT ; dst_vma =find_dst_vma (dst_mm, start, len); -/* -* Make sure the vma is not shared, that the dst range is -* both valid and fully within a single existing vma. -*/ -if (!dst_vma || (dst_vma->vm_flags &VM_SHARED )) + +if (!dst_vma) goto out_unlock;if (!userfaultfd_wp (dst_vma)) goto out_unlock; -if (!vma_is_anonymous (dst_vma)) +if (!vma_can_userfault (dst_vma, dst_vma->vm_flags)) goto out_unlock;if (is_vm_hugetlb_page (dst_vma)) {
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 22/23 mm: Enable PTE markers by default
In reply to Peter XuEnable PTE markers by default. On x86_64 it means it'll auto-enable PTE_MARKER_UFFD_WP as well.
--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/Kconfig +1 -1
@@ -901,7 +901,7 @@ config ANON_VMA_NAME difference in their name. config PTE_MARKER - def_bool n + def_bool y bool "Marker PTEs support" help
Signature · 1 line
2.32.0
Peter Xu
reviewerPATCH v7 23/23 selftests/uffd: Enable uffd-wp for shmem/hugetlbfs
In reply to Peter XuAfter we added support for shmem and hugetlbfs, we can turn uffd-wp test on always now.
--- tools/testing/selftests/vm/userfaultfd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
tools/testing/selftests/vm/userfaultfd.c +1 -3
@@ -82,7 +82,7 @@ static int test_type;static volatile bool test_uffdio_copy_eexist = true;static volatile bool test_uffdio_zeropage_eexist = true;/* Whether to test uffd write-protection */ -static bool test_uffdio_wp = false; +static bool test_uffdio_wp = true;/* Whether to test uffd minor faults */ static bool test_uffdio_minor = false;
@@ -1597,8 +1597,6 @@ static void set_test_type(const char *type)if (!strcmp (type,"anon" )) { test_type =TEST_ANON ; uffd_test_ops = &anon_uffd_test_ops; -/* Only enable write-protect test for anonymous test */ - test_uffdio_wp = true; }else if (!strcmp (type,"hugetlb" )) { test_type =TEST_HUGETLB ; uffd_test_ops = &hugetlb_uffd_test_ops;
Signature · 1 line
2.32.0