mm/hugetlb: Make huge_pte_offset() thread-safe for pmd unshare
Subsystems (8)
v3 cover mm/hugetlb: Make huge_pte_offset() thread-safe for pmd unshare
Based on latest mm-unstable (dbccafc6e992).
This can be seen as a follow-up series to Mike's recent hugetlb vma lock series for pmd unsharing, but majorly covering safe use of huge_pte_offset.
Changelog for v2->v3:
- Added a-bs and r-bs
- In walk_hugetlb_range() hooks, don't release vma lock for cond_resched()
- Applied quite a few John's comment suggestions in multiple patches
- Squashed patch 10 into patch 9
Old versions:
rfcv1: https://lore.kernel.org/r/[email protected] rfcv2: https://lore.kernel.org/r/[email protected] v1: https://lore.kernel.org/r/[email protected] v2: https://lore.kernel.org/r/[email protected]
Problem
=======
huge_pte_offset() is a major helper used by hugetlb code paths to walk a hugetlb pgtable. It's used mostly everywhere since that's needed even before taking the pgtable lock.
huge_pte_offset() is always called with mmap lock held with either read or write. It was assumed to be safe but it's actually not. One race condition can easily trigger by: (1) firstly trigger pmd share on a memory range, (2) do huge_pte_offset() on the range, then at the meantime, (3) another thread unshare the pmd range, and the pgtable page is prone to lost if the other shared process wants to free it completely (by either munmap or exit mm).
The recent work from Mike on vma lock can resolve most of this already. It's achieved by forbidden pmd unsharing during the lock being taken, so no further risk of the pgtable page being freed. It means if we can take the vma lock around all huge_pte_offset() callers it'll be safe.
There're already a bunch of them that we did as per the latest mm-unstable, but also quite a few others that we didn't for various reasons especially on huge_pte_offset() usage.
One more thing to mention is that besides the vma lock, i_mmap_rwsem can also be used to protect the pgtable page (along with its pgtable lock) from being freed from under us. IOW, huge_pte_offset() callers need to either hold the vma lock or i_mmap_rwsem to safely walk the pgtables.
A reproducer of such problem, based on hugetlb GUP (NOTE: since the race is very hard to trigger, one needs to apply another kernel delay patch too, see below):
======8<=======
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <linux/memfd.h>
#include <assert.h>
#include <pthread.h>
#define MSIZE (1UL << 30) /* 1GB */ #define PSIZE (2UL << 20) /* 2MB */
#define HOLD_SEC (1)
int pipefd[2];
void *buf;
void *do_map(int fd)
{
unsigned char *tmpbuf, *p;
int ret; ret = posix_memalign((void **)&tmpbuf, MSIZE, MSIZE);
if (ret) {
perror("posix_memalign() failed");
return NULL;
}
tmpbuf = mmap(tmpbuf, MSIZE, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_FIXED, fd, 0);
if (tmpbuf == MAP_FAILED) {
perror("mmap() failed");
return NULL;
}
printf("mmap() -> %p\n", tmpbuf);
for (p = tmpbuf; p < tmpbuf + MSIZE; p += PSIZE) {
*p = 1;
}
return tmpbuf;}
void do_unmap(void *buf)
{
munmap(buf, MSIZE);
}
void proc2(int fd)
{
unsigned char c; buf = do_map(fd);
if (!buf)
return;
read(pipefd[0], &c, 1);
/*
* This frees the shared pgtable page, causing use-after-free in
* proc1_thread1 when soft walking hugetlb pgtable.
*/
do_unmap(buf);
printf("Proc2 quitting\n");}
void *proc1_thread1(void *data)
{
/*
* Trigger follow-page on 1st 2m page. Kernel hack patch needed to
* withhold this procedure for easier reproduce.
*/
madvise(buf, PSIZE, MADV_POPULATE_WRITE);
printf("Proc1-thread1 quitting\n");
return NULL;
}
void *proc1_thread2(void *data)
{
unsigned char c; /* Wait a while until proc1_thread1() start to wait */
sleep(0.5);
/* Trigger pmd unshare */
madvise(buf, PSIZE, MADV_DONTNEED);
/* Kick off proc2 to release the pgtable */
write(pipefd[1], &c, 1);
printf("Proc1-thread2 quitting\n");
return NULL;}
void proc1(int fd)
{
pthread_t tid1, tid2;
int ret; buf = do_map(fd);
if (!buf)
return;
ret = pthread_create(&tid1, NULL, proc1_thread1, NULL);
assert(ret == 0);
ret = pthread_create(&tid2, NULL, proc1_thread2, NULL);
assert(ret == 0);
/* Kick the child to share the PUD entry */
pthread_join(tid1, NULL);
pthread_join(tid2, NULL);
do_unmap(buf);}
int main(void)
{
int fd, ret; fd = memfd_create("test-huge", MFD_HUGETLB | MFD_HUGE_2MB);
if (fd < 0) {
perror("open failed");
return -1;
}
ret = ftruncate(fd, MSIZE);
if (ret) {
perror("ftruncate() failed");
return -1;
}
ret = pipe(pipefd);
if (ret) {
perror("pipe() failed");
return -1;
}
if (fork()) {
proc1(fd);
} else {
proc2(fd);
}
close(pipefd[0]);
close(pipefd[1]);
close(fd);
return 0; }
======8<=======
The kernel patch needed to present such a race so it'll trigger 100%:
======8<=======
mm/hugetlb.c +9 -0
@@ -38,6 +38,7 @@#include <asm/page.h> #include <asm/pgalloc.h> #include <asm/tlb.h> +#include <asm/delay.h> #include <linux/io.h> #include <linux/hugetlb.h>
@@ -6290,6 +6291,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,bool unshare = false;int absent;struct page *page; +unsigned long c =0 ;/* * If we have a pending SIGKILL, don't keep faulting pages and
@@ -6309,6 +6311,13 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,*/ pte =huge_pte_offset (mm, vaddr &huge_page_mask (h),huge_page_size (h)); + +pr_info ("%s: withhold 1 sec...\n" , __func__); +for (c =0 ; c <100 ; c++) { +udelay (10000 ); + } +pr_info ("%s: withhold 1 sec...done\n" , __func__); +if (pte) ptl =huge_pte_lock (h, mm, pte); absent = !pte ||huge_pte_none (huge_ptep_get (pte));
======8<=======
It'll trigger use-after-free of the pgtable spinlock:
======8<=======
[ 16.959907] follow_hugetlb_page: withhold 1 sec... [ 17.960315] follow_hugetlb_page: withhold 1 sec...done [ 17.960550] ------------[ cut here ]------------ [ 17.960742] DEBUG_LOCKS_WARN_ON(1) [ 17.960756] WARNING: CPU: 3 PID: 542 at kernel/locking/lockdep.c:231 __lock_acquire+0x955/0x1fa0 [ 17.961264] Modules linked in: [ 17.961394] CPU: 3 PID: 542 Comm: hugetlb-pmd-sha Not tainted 6.1.0-rc4-peterx+ #46 [ 17.961704] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 17.962266] RIP: 0010:__lock_acquire+0x955/0x1fa0 [ 17.962516] Code: c0 0f 84 5f fe ff ff 44 8b 1d 0f 9a 29 02 45 85 db 0f 85 4f fe ff ff 48 c7 c6 75 50 83 82 48 c7 c7 1b 4b 7d 82 e8 d3 22 d8 00 <0f> 0b 31 c0 4c 8b 54 24 08 4c 8b 04 24 e9 [ 17.963494] RSP: 0018:ffffc90000e4fba8 EFLAGS: 00010096 [ 17.963704] RAX: 0000000000000016 RBX: fffffffffd3925a8 RCX: 0000000000000000 [ 17.963989] RDX: 0000000000000002 RSI: ffffffff82863ccf RDI: 00000000ffffffff [ 17.964276] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffc90000e4fa58 [ 17.964557] R10: 0000000000000003 R11: ffffffff83162688 R12: 0000000000000000 [ 17.964839] R13: 0000000000000001 R14: ffff888105eac748 R15: 0000000000000001 [ 17.965123] FS: 00007f17c0a00640(0000) GS:ffff888277cc0000(0000) knlGS:0000000000000000 [ 17.965443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 17.965672] CR2: 00007f17c09ffef8 CR3: 000000010c87a005 CR4: 0000000000770ee0 [ 17.965956] PKRU: 55555554 [ 17.966068] Call Trace: [ 17.966172] <TASK> [ 17.966268] ? tick_nohz_tick_stopped+0x12/0x30 [ 17.966455] lock_acquire+0xbf/0x2b0 [ 17.966603] ? follow_hugetlb_page.cold+0x75/0x5c4 [ 17.966799] ? _printk+0x48/0x4e [ 17.966934] _raw_spin_lock+0x2f/0x40 [ 17.967087] ? follow_hugetlb_page.cold+0x75/0x5c4 [ 17.967285] follow_hugetlb_page.cold+0x75/0x5c4 [ 17.967473] __get_user_pages+0xbb/0x620 [ 17.967635] faultin_vma_page_range+0x9a/0x100 [ 17.967817] madvise_vma_behavior+0x3c0/0xbd0 [ 17.967998] ? mas_prev+0x11/0x290 [ 17.968141] ? find_vma_prev+0x5e/0xa0 [ 17.968304] ? madvise_vma_anon_name+0x70/0x70 [ 17.968486] madvise_walk_vmas+0xa9/0x120 [ 17.968650] do_madvise.part.0+0xfa/0x270 [ 17.968813] __x64_sys_madvise+0x5a/0x70 [ 17.968974] do_syscall_64+0x37/0x90 [ 17.969123] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 17.969329] RIP: 0033:0x7f1840f0efdb [ 17.969477] Code: c3 66 0f 1f 44 00 00 48 8b 15 39 6e 0e 00 f7 d8 64 89 02 b8 ff ff ff ff eb bc 0f 1f 44 00 00 f3 0f 1e fa b8 1c 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d 68 [ 17.970205] RSP: 002b:00007f17c09ffe38 EFLAGS: 00000202 ORIG_RAX: 000000000000001c [ 17.970504] RAX: ffffffffffffffda RBX: 00007f17c0a00640 RCX: 00007f1840f0efdb [ 17.970786] RDX: 0000000000000017 RSI: 0000000000200000 RDI: 00007f1800000000 [ 17.971068] RBP: 00007f17c09ffe50 R08: 0000000000000000 R09: 00007ffd3954164f [ 17.971353] R10: 00007f1840e10348 R11: 0000000000000202 R12: ffffffffffffff80 [ 17.971709] R13: 0000000000000000 R14: 00007ffd39541550 R15: 00007f17c0200000 [ 17.972083] </TASK> [ 17.972199] irq event stamp: 2353 [ 17.972372] hardirqs last enabled at (2353): [<ffffffff8117fe4e>] __up_console_sem+0x5e/0x70 [ 17.972869] hardirqs last disabled at (2352): [<ffffffff8117fe33>] __up_console_sem+0x43/0x70 [ 17.973365] softirqs last enabled at (2330): [<ffffffff810f763d>] __irq_exit_rcu+0xed/0x160 [ 17.973857] softirqs last disabled at (2323): [<ffffffff810f763d>] __irq_exit_rcu+0xed/0x160 [ 17.974341] ---[ end trace 0000000000000000 ]--- [ 17.974614] BUG: kernel NULL pointer dereference, address: 00000000000000b8 [ 17.975012] #PF: supervisor read access in kernel mode [ 17.975314] #PF: error_code(0x0000) - not-present page [ 17.975615] PGD 103f7b067 P4D 103f7b067 PUD 106cd7067 PMD 0 [ 17.975943] Oops: 0000 [#1] PREEMPT SMP NOPTI [ 17.976197] CPU: 3 PID: 542 Comm: hugetlb-pmd-sha Tainted: G W 6.1.0-rc4-peterx+ #46 [ 17.976712] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 17.977370] RIP: 0010:__lock_acquire+0x190/0x1fa0 [ 17.977655] Code: 98 00 00 00 41 89 46 24 81 e2 ff 1f 00 00 48 0f a3 15 e4 ba dd 02 0f 83 ff 05 00 00 48 8d 04 52 48 c1 e0 06 48 05 c0 d2 f4 83 <44> 0f b6 a0 b8 00 00 00 41 0f b7 46 20 6f [ 17.979170] RSP: 0018:ffffc90000e4fba8 EFLAGS: 00010046 [ 17.979787] RAX: 0000000000000000 RBX: fffffffffd3925a8 RCX: 0000000000000000 [ 17.980838] RDX: 0000000000000002 RSI: ffffffff82863ccf RDI: 00000000ffffffff [ 17.982048] RBP: 0000000000000000 R08: ffff888105eac720 R09: ffffc90000e4fa58 [ 17.982892] R10: ffff888105eab900 R11: ffffffff83162688 R12: 0000000000000000 [ 17.983771] R13: 0000000000000001 R14: ffff888105eac748 R15: 0000000000000001 [ 17.984815] FS: 00007f17c0a00640(0000) GS:ffff888277cc0000(0000) knlGS:0000000000000000 [ 17.985924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 17.986265] CR2: 00000000000000b8 CR3: 000000010c87a005 CR4: 0000000000770ee0 [ 17.986674] PKRU: 55555554 [ 17.986832] Call Trace: [ 17.987012] <TASK> [ 17.987266] ? tick_nohz_tick_stopped+0x12/0x30 [ 17.987770] lock_acquire+0xbf/0x2b0 [ 17.988118] ? follow_hugetlb_page.cold+0x75/0x5c4 [ 17.988575] ? _printk+0x48/0x4e [ 17.988889] _raw_spin_lock+0x2f/0x40 [ 17.989243] ? follow_hugetlb_page.cold+0x75/0x5c4 [ 17.989687] follow_hugetlb_page.cold+0x75/0x5c4 [ 17.990119] __get_user_pages+0xbb/0x620 [ 17.990500] faultin_vma_page_range+0x9a/0x100 [ 17.990928] madvise_vma_behavior+0x3c0/0xbd0 [ 17.991354] ? mas_prev+0x11/0x290 [ 17.991678] ? find_vma_prev+0x5e/0xa0 [ 17.992024] ? madvise_vma_anon_name+0x70/0x70 [ 17.992421] madvise_walk_vmas+0xa9/0x120 [ 17.992793] do_madvise.part.0+0xfa/0x270 [ 17.993166] __x64_sys_madvise+0x5a/0x70 [ 17.993539] do_syscall_64+0x37/0x90 [ 17.993879] entry_SYSCALL_64_after_hwframe+0x63/0xcd
======8<=======
Resolution
==========
This patchset protects all the huge_pte_offset() callers to also take the vma lock properly.
Patch Layout
============
Patch 1-2: cleanup, or dependency of the follow up patches Patch 3: before fixing, document huge_pte_offset() on lock required Patch 4-8: each patch resolves one possible race condition Patch 9: introduce hugetlb_walk() to replace huge_pte_offset() Patch 10: document why page_vma_mapped_walk() is safe
Tests
=====
The series is verified with the above reproducer so the race cannot trigger anymore. It also passes all hugetlb kselftests.
Comments welcomed, thanks.
Peter Xu (9):
mm/hugetlb: Let vma_offset_start() to return start mm/hugetlb: Don't wait for migration entry during follow page mm/hugetlb: Document huge_pte_offset usage
mm/hugetlb: Move swap entry handling into vma lock when faulted mm/hugetlb: Make userfaultfd_huge_must_wait() safe to pmd unshare mm/hugetlb: Make hugetlb_follow_page_mask() safe to pmd unshare mm/hugetlb: Make follow_hugetlb_page() safe to pmd unshare mm/hugetlb: Make walk_hugetlb_range() safe to pmd unshare mm/hugetlb: Introduce hugetlb_walk()
fs/hugetlbfs/inode.c | 28 ++++++------ fs/userfaultfd.c | 26 ++++++++---- include/linux/hugetlb.h | 71 +++++++++++++++++++++++++++++++ include/linux/pagewalk.h | 11 ++++- include/linux/swapops.h | 6 ++- mm/hmm.c | 15 ++++++- mm/hugetlb.c | 92 +++++++++++++++++----------------------- mm/migrate.c | 25 +++++++++-- mm/page_vma_mapped.c | 9 ++-- mm/pagewalk.c | 6 +-- 10 files changed, 200 insertions(+), 89 deletions(-)
Signature · 1 line
2.37.3
PATCH v3 1/9 mm/hugetlb: Let vma_offset_start() to return start
Even though vma_offset_start() is named like that, it's not returning "the start address of the range" but rather the offset we should use to offset the vma->vm_start address.
Make it return the real value of the start vaddr, and it also helps for all the callers because whenever the retval is used, it'll be ultimately added into the vma->vm_start anyway, so it's better.
--- fs/hugetlbfs/inode.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
fs/hugetlbfs/inode.c +12 -12
@@ -412,10 +412,12 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma,*/ static unsigned long vma_offset_start (struct vm_area_struct *vma,pgoff_t start) { +unsigned long offset =0 ; +if (vma->vm_pgoff < start) -return (start - vma->vm_pgoff) <<PAGE_SHIFT ; -else -return 0 ; + offset = (start - vma->vm_pgoff) <<PAGE_SHIFT ; + +return vma->vm_start + offset; }static unsigned long vma_offset_end (struct vm_area_struct *vma,pgoff_t end)
@@ -457,7 +459,7 @@ static void hugetlb_unmap_file_folio(struct hstate *h, v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -if (!hugetlb_vma_maps_page (vma, vma->vm_start + v_start, page)) +if (!hugetlb_vma_maps_page (vma, v_start, page))continue ;if (!hugetlb_vma_trylock_write (vma)) {
@@ -473,8 +475,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h,break ; } -unmap_hugepage_range (vma, vma->vm_start + v_start, v_end, -NULL ,ZAP_FLAG_DROP_MARKER ); +unmap_hugepage_range (vma, v_start, v_end,NULL , +ZAP_FLAG_DROP_MARKER );hugetlb_vma_unlock_write (vma); }
@@ -507,10 +509,9 @@ static void hugetlb_unmap_file_folio(struct hstate *h,*/ v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -if (hugetlb_vma_maps_page (vma, vma->vm_start + v_start, page)) -unmap_hugepage_range (vma, vma->vm_start + v_start, - v_end,NULL , -ZAP_FLAG_DROP_MARKER ); +if (hugetlb_vma_maps_page (vma, v_start, page)) +unmap_hugepage_range (vma, v_start, v_end,NULL , +ZAP_FLAG_DROP_MARKER );kref_put (&vma_lock->refs, hugetlb_vma_lock_release);hugetlb_vma_unlock_write (vma);
@@ -540,8 +541,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -unmap_hugepage_range (vma, vma->vm_start + v_start, v_end, -NULL , zap_flags); +unmap_hugepage_range (vma, v_start, v_end,NULL , zap_flags);/* * Note that vma lock only exists for shared/non-private
Signature · 1 line
2.37.3
PATCH v3 2/9 mm/hugetlb: Don't wait for migration entry during follow page
In reply to Peter XuThat's what the code does with !hugetlb pages, so we should logically do the same for hugetlb, so migration entry will also be treated as no page.
This is probably also the last piece in follow_page code that may sleep, the last one should be removed in cf994dd8af27 ("mm/gup: remove FOLL_MIGRATION", 2022-11-16).
--- mm/hugetlb.c | 11 ----------- 1 file changed, 11 deletions(-)
mm/hugetlb.c +0 -11
@@ -6232,7 +6232,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,if (WARN_ON_ONCE (flags &FOLL_PIN ))return NULL ; -retry: pte =huge_pte_offset (mm, haddr,huge_page_size (h));if (!pte)return NULL ;
@@ -6255,16 +6254,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page =NULL ; goto out; } - }else { -if (is_hugetlb_entry_migration (entry)) { -spin_unlock (ptl); -__migration_entry_wait_huge (pte, ptl); - goto retry; - } -/* -* hwpoisoned entry is treated as no_page_table in -* follow_page_mask(). -*/ } out:spin_unlock (ptl);
Signature · 1 line
2.37.3
PATCH v3 3/9 mm/hugetlb: Document huge_pte_offset usage
In reply to Peter Xuhuge_pte_offset() is potentially a pgtable walker, looking up pte_t* for a hugetlb address.
Normally, it's always safe to walk a generic pgtable as long as we're with the mmap lock held for either read or write, because that guarantees the pgtable pages will always be valid during the process.
But it's not true for hugetlbfs, especially shared: hugetlbfs can have its pgtable freed by pmd unsharing, it means that even with mmap lock held for current mm, the PMD pgtable page can still go away from under us if pmd unsharing is possible during the walk.
So we have two ways to make it safe even for a shared mapping:
(1) If we're with the hugetlb vma lock held for either read/write, it's okay because pmd unshare cannot happen at all.
(2) If we're with the i_mmap_rwsem lock held for either read/write, it's okay because even if pmd unshare can happen, the pgtable page cannot be freed from under us.
Document it.
--- include/linux/hugetlb.h | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
include/linux/hugetlb.h +32 -0
@@ -192,6 +192,38 @@ extern struct list_head huge_boot_pages;pte_t *huge_pte_alloc (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,unsigned long sz); +/* +* huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. +* Returns the pte_t* if found, or NULL if the address is not mapped. +* +* Since this function will walk all the pgtable pages (including not only +* high-level pgtable page, but also PUD entry that can be unshared +* concurrently for VM_SHARED), the caller of this function should be +* responsible of its thread safety. One can follow this rule: +* +* (1) For private mappings: pmd unsharing is not possible, so holding the +* mmap_lock for either read or write is sufficient. Most callers +* already hold the mmap_lock, so normally, no special action is +* required. +* +* (2) For shared mappings: pmd unsharing is possible (so the PUD-ranged +* pgtable page can go away from under us! It can be done by a pmd +* unshare with a follow up munmap() on the other process), then we +* need either: +* +* (2.1) hugetlb vma lock read or write held, to make sure pmd unshare +* won't happen upon the range (it also makes sure the pte_t we +* read is the right and stable one), or, +* +* (2.2) hugetlb mapping i_mmap_rwsem lock held read or write, to make +* sure even if unshare happened the racy unmap() will wait until +* i_mmap_rwsem is released. +* +* Option (2.1) is the safest, which guarantees pte stability from pmd +* sharing pov, until the vma lock released. Option (2.2) doesn't protect +* a concurrent pmd unshare, but it makes sure the pgtable page is safe to +* access. +*/ pte_t *huge_pte_offset (struct mm_struct *mm,unsigned long addr,unsigned long sz);unsigned long hugetlb_mask_last_page (struct hstate *h);
Signature · 1 line
2.37.3
PATCH v3 4/9 mm/hugetlb: Move swap entry handling into vma lock when faulted
In reply to Peter XuIn hugetlb_fault(), there used to have a special path to handle swap entry at the entrance using huge_pte_offset(). That's unsafe because huge_pte_offset() for a pmd sharable range can access freed pgtables if without any lock to protect the pgtable from being freed after pmd unshare.
Here the simplest solution to make it safe is to move the swap handling to be after the vma lock being held. We may need to take the fault mutex on either migration or hwpoison entries now (also the vma lock, but that's really needed), however neither of them is hot path.
Note that the vma lock cannot be released in hugetlb_fault() when the migration entry is detected, because in migration_entry_wait_huge() the pgtable page will be used again (by taking the pgtable lock), so that also need to be protected by the vma lock. Modify migration_entry_wait_huge() so that it must be called with vma read lock held, and properly release the lock in __migration_entry_wait_huge().
--- include/linux/swapops.h | 6 ++++-- mm/hugetlb.c | 37 ++++++++++++++++--------------------- mm/migrate.c | 25 +++++++++++++++++++++---- 3 files changed, 41 insertions(+), 27 deletions(-)
include/linux/swapops.h +4 -2
@@ -337,7 +337,8 @@ extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep,extern void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address);#ifdef CONFIG_HUGETLB_PAGE -extern void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl); +extern void __migration_entry_wait_huge (struct vm_area_struct *vma, +pte_t *ptep,spinlock_t *ptl);extern void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte);#endif /* CONFIG_HUGETLB_PAGE */ #else /* CONFIG_MIGRATION */
@@ -366,7 +367,8 @@ static inline void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep,static inline void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address) { }#ifdef CONFIG_HUGETLB_PAGE -static inline void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl) { } +static inline void __migration_entry_wait_huge (struct vm_area_struct *vma, +pte_t *ptep,spinlock_t *ptl) { }static inline void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) { }#endif /* CONFIG_HUGETLB_PAGE */ static inline int is_writable_migration_entry (swp_entry_t entry)
mm/hugetlb.c +16 -21
@@ -5824,22 +5824,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,int need_wait_lock =0 ;unsigned long haddr = address &huge_page_mask (h); - ptep =huge_pte_offset (mm, haddr,huge_page_size (h)); -if (ptep) { -/* -* Since we hold no locks, ptep could be stale. That is -* OK as we are only making decisions based on content and -* not actually modifying content here. -*/ - entry =huge_ptep_get (ptep); -if (unlikely (is_hugetlb_entry_migration (entry))) { -migration_entry_wait_huge (vma, ptep); -return 0 ; - }else if (unlikely (is_hugetlb_entry_hwpoisoned (entry))) -return VM_FAULT_HWPOISON_LARGE | -VM_FAULT_SET_HINDEX (hstate_index (h)); - } -/* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate
@@ -5854,10 +5838,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,* Acquire vma lock before calling huge_pte_alloc and hold * until finished with ptep. This prevents huge_pmd_unshare from * being called elsewhere and making the ptep no longer valid. -* -* ptep could have already be assigned via huge_pte_offset. That -* is OK, as huge_pte_alloc will return the same value unless -* something has changed. */ hugetlb_vma_lock_read (vma); ptep =huge_pte_alloc (mm, vma, haddr,huge_page_size (h));
@@ -5886,8 +5866,23 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,* fault, and is_hugetlb_entry_(migration|hwpoisoned) check will * properly handle it. */ -if (!pte_present (entry)) +if (!pte_present (entry)) { +if (unlikely (is_hugetlb_entry_migration (entry))) { +/* +* Release the hugetlb fault lock now, but retain +* the vma lock, because it is needed to guard the +* huge_pte_lockptr() later in +* migration_entry_wait_huge(). The vma lock will +* be released there. +*/ +mutex_unlock (&hugetlb_fault_mutex_table[hash]); +migration_entry_wait_huge (vma, ptep); +return 0 ; + }else if (unlikely (is_hugetlb_entry_hwpoisoned (entry))) + ret =VM_FAULT_HWPOISON_LARGE | +VM_FAULT_SET_HINDEX (hstate_index (h)); goto out_mutex; + }/* * If we are going to COW/unshare the mapping later, we examine the
mm/migrate.c +21 -4
@@ -333,24 +333,41 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, }#ifdef CONFIG_HUGETLB_PAGE -void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl) +/* +* The vma read lock must be held upon entry. Holding that lock prevents either +* the pte or the ptl from being freed. +* +* This function will release the vma lock before returning. +*/ +void __migration_entry_wait_huge (struct vm_area_struct *vma, +pte_t *ptep,spinlock_t *ptl) {pte_t pte; +hugetlb_vma_assert_locked (vma);spin_lock (ptl); pte =huge_ptep_get (ptep); -if (unlikely (!is_hugetlb_entry_migration (pte))) +if (unlikely (!is_hugetlb_entry_migration (pte))) {spin_unlock (ptl); -else +hugetlb_vma_unlock_read (vma); + }else { +/* +* If migration entry existed, safe to release vma lock +* here because the pgtable page won't be freed without the +* pgtable lock released. See comment right above pgtable +* lock release in migration_entry_wait_on_locked(). +*/ +hugetlb_vma_unlock_read (vma);migration_entry_wait_on_locked (pte_to_swp_entry (pte),NULL , ptl); + } }void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) {spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), vma->vm_mm, pte); -__migration_entry_wait_huge (pte, ptl); +__migration_entry_wait_huge (vma, pte, ptl); }#endif
Signature · 1 line
2.37.3
PATCH v3 5/9 mm/hugetlb: Make userfaultfd_huge_must_wait() safe to pmd unshare
In reply to Peter XuWe can take the hugetlb walker lock, here taking vma lock directly.
--- fs/userfaultfd.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
fs/userfaultfd.c +16 -4
@@ -376,7 +376,8 @@ static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags)*/ vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason) { -struct mm_struct *mm = vmf->vma->vm_mm; +struct vm_area_struct *vma = vmf->vma; +struct mm_struct *mm = vma->vm_mm;struct userfaultfd_ctx *ctx;struct userfaultfd_wait_queue uwq;vm_fault_t ret =VM_FAULT_SIGBUS ;
@@ -403,7 +404,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)*/ mmap_assert_locked (mm); - ctx = vmf->vma->vm_userfaultfd_ctx.ctx; + ctx = vma->vm_userfaultfd_ctx.ctx;if (!ctx) goto out;
@@ -493,6 +494,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) blocking_state =userfaultfd_get_blocking_state (vmf->flags); +/* +* Take the vma lock now, in order to safely call +* userfaultfd_huge_must_wait() later. Since acquiring the +* (sleepable) vma lock can modify the current task state, that +* must be before explicitly calling set_current_state(). +*/ +if (is_vm_hugetlb_page (vma)) +hugetlb_vma_lock_read (vma); +spin_lock_irq (&ctx->fault_pending_wqh.lock);/* * After the __add_wait_queue the uwq is visible to userland
@@ -507,13 +517,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)set_current_state (blocking_state);spin_unlock_irq (&ctx->fault_pending_wqh.lock); -if (!is_vm_hugetlb_page (vmf->vma)) +if (!is_vm_hugetlb_page (vma)) must_wait =userfaultfd_must_wait (ctx, vmf->address, vmf->flags, reason);else - must_wait =userfaultfd_huge_must_wait (ctx, vmf->vma, + must_wait =userfaultfd_huge_must_wait (ctx, vma, vmf->address, vmf->flags, reason); +if (is_vm_hugetlb_page (vma)) +hugetlb_vma_unlock_read (vma);mmap_read_unlock (mm);if (likely (must_wait && !READ_ONCE (ctx->released))) {
Signature · 1 line
2.37.3
PATCH v3 6/9 mm/hugetlb: Make hugetlb_follow_page_mask() safe to pmd unshare
In reply to Peter XuSince hugetlb_follow_page_mask() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently.
--- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/hugetlb.c +4 -1
@@ -6227,9 +6227,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,if (WARN_ON_ONCE (flags &FOLL_PIN ))return NULL ; +hugetlb_vma_lock_read (vma); pte =huge_pte_offset (mm, haddr,huge_page_size (h));if (!pte) -return NULL ; + goto out_unlock; ptl =huge_pte_lock (h, mm, pte); entry =huge_ptep_get (pte);
@@ -6252,6 +6253,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, } out:spin_unlock (ptl); +out_unlock: +hugetlb_vma_unlock_read (vma);return page; }
Signature · 1 line
2.37.3
PATCH v3 7/9 mm/hugetlb: Make follow_hugetlb_page() safe to pmd unshare
In reply to Peter XuSince follow_hugetlb_page() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently.
--- mm/hugetlb.c | 7 +++++++ 1 file changed, 7 insertions(+)
mm/hugetlb.c +7 -0
@@ -6285,6 +6285,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,break ; } +hugetlb_vma_lock_read (vma);/* * Some archs (sparc64, sh*) have multiple pte_ts to * each hugepage. We have to make sure we get the
@@ -6309,6 +6310,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, !hugetlbfs_pagecache_present(h, vma, vaddr)) {if (pte)spin_unlock (ptl); +hugetlb_vma_unlock_read (vma); remainder =0 ;break ; }
@@ -6330,6 +6332,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,if (pte)spin_unlock (ptl); +hugetlb_vma_unlock_read (vma); +if (flags &FOLL_WRITE ) fault_flags |=FAULT_FLAG_WRITE ;else if (unshare)
@@ -6389,6 +6393,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, remainder -=pages_per_huge_page (h); i +=pages_per_huge_page (h);spin_unlock (ptl); +hugetlb_vma_unlock_read (vma);continue ; }
@@ -6416,6 +6421,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,if (WARN_ON_ONCE (!try_grab_folio (pages[i], refs, flags))) {spin_unlock (ptl); +hugetlb_vma_unlock_read (vma); remainder =0 ; err = -ENOMEM ;break ;
@@ -6427,6 +6433,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, i += refs;spin_unlock (ptl); +hugetlb_vma_unlock_read (vma); } *nr_pages = remainder;/*
Signature · 1 line
2.37.3
PATCH v3 8/9 mm/hugetlb: Make walk_hugetlb_range() safe to pmd unshare
In reply to Peter XuSince walk_hugetlb_range() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently.
--- include/linux/pagewalk.h | 11 ++++++++++- mm/hmm.c | 15 ++++++++++++++- mm/pagewalk.c | 2 ++ 3 files changed, 26 insertions(+), 2 deletions(-)
include/linux/pagewalk.h +10 -1
@@ -21,7 +21,16 @@ struct mm_walk;* depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD. * Any folded depths (where PTRS_PER_P?D is equal to 1) * are skipped. -* @hugetlb_entry: if set, called for each hugetlb entry +* @hugetlb_entry: if set, called for each hugetlb entry. This hook +* function is called with the vma lock held, in order to +* protect against a concurrent freeing of the pte_t* or +* the ptl. In some cases, the hook function needs to drop +* and retake the vma lock in order to avoid deadlocks +* while calling other functions. In such cases the hook +* function must either refrain from accessing the pte or +* ptl after dropping the vma lock, or else revalidate +* those items after re-acquiring the vma lock and before +* accessing them. * @test_walk: caller specific callback function to determine whether * we walk over the current vma or not. Returning 0 means * "do page table walk over the current vma", returning
mm/hmm.c +14 -1
@@ -493,8 +493,21 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, required_fault =hmm_pte_need_fault (hmm_vma_walk, pfn_req_flags, cpu_flags);if (required_fault) { +int ret; +spin_unlock (ptl); -return hmm_vma_fault (addr, end, required_fault, walk); +hugetlb_vma_unlock_read (vma); +/* +* Avoid deadlock: drop the vma lock before calling +* hmm_vma_fault(), which will itself potentially take and +* drop the vma lock. This is also correct from a +* protection point of view, because there is no further +* use here of either pte or ptl after dropping the vma +* lock. +*/ + ret =hmm_vma_fault (addr, end, required_fault, walk); +hugetlb_vma_lock_read (vma); +return ret; } pfn =pte_pfn (entry) + ((start & ~hmask) >>PAGE_SHIFT );
mm/pagewalk.c +2 -0
@@ -302,6 +302,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,const struct mm_walk_ops *ops = walk->ops;int err =0 ; +hugetlb_vma_lock_read (vma); do { next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask, sz);
@@ -314,6 +315,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,if (err)break ; }while (addr = next, addr != end); +hugetlb_vma_unlock_read (vma);return err; }
Signature · 1 line
2.37.3
Re: [PATCH v3 8/9] mm/hugetlb: Make walk_hugetlb_range() safe to pmd unshare
Quoted 10 lines · Peter Xu
Since
walk_hugetlb_range()walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently.Reviewed-by Mike Kravetz <[email protected]>Signed-off-by Peter Xu <[email protected]>--- include/linux/pagewalk.h | 11 ++++++++++- mm/hmm.c | 15 ++++++++++++++- mm/pagewalk.c | 2 ++ 3 files changed, 26 insertions(+), 2 deletions(-)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 70 lines
include/linux/pagewalk.h +10
-1@@ -21,7 +21,16 @@ struct mm_walk;* depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD. * Any folded depths (where PTRS_PER_P?D is equal to 1) * are skipped. -* @hugetlb_entry: if set, called for each hugetlb entry +* @hugetlb_entry: if set, called for each hugetlb entry. This hook +* function is called with the vma lock held, in order to +* protect against a concurrent freeing of the pte_t* or +* the ptl. In some cases, the hook function needs to drop +* and retake the vma lock in order to avoid deadlocks +* while calling other functions. In such cases the hook +* function must either refrain from accessing the pte or +* ptl after dropping the vma lock, or else revalidate +* those items after re-acquiring the vma lock and before +* accessing them. * @test_walk: caller specific callback function to determine whether * we walk over the current vma or not. Returning 0 means * "do page table walk over the current vma", returning mm/hmm.c +14
-1@@ -493,8 +493,21 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, required_fault =hmm_pte_need_fault (hmm_vma_walk, pfn_req_flags, cpu_flags);if (required_fault) { +int ret; +spin_unlock (ptl); -return hmm_vma_fault (addr, end, required_fault, walk); +hugetlb_vma_unlock_read (vma); +/* +* Avoid deadlock: drop the vma lock before calling +* hmm_vma_fault(), which will itself potentially take and +* drop the vma lock. This is also correct from a +* protection point of view, because there is no further +* use here of either pte or ptl after dropping the vma +* lock. +*/ + ret =hmm_vma_fault (addr, end, required_fault, walk); +hugetlb_vma_lock_read (vma); +return ret; } pfn =pte_pfn (entry) + ((start & ~hmask) >>PAGE_SHIFT );mm/pagewalk.c +2
-0@@ -302,6 +302,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,const struct mm_walk_ops *ops = walk->ops;int err =0 ; +hugetlb_vma_lock_read (vma); do { next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask, sz);@@ -314,6 +315,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,if (err)break ; }while (addr = next, addr != end); +hugetlb_vma_unlock_read (vma);return err; }
PATCH v3 9/9 mm/hugetlb: Introduce hugetlb_walk()
In reply to Peter Xuhuge_pte_offset() is the main walker function for hugetlb pgtables. The name is not really representing what it does, though.
Instead of renaming it, introduce a wrapper function called hugetlb_walk() which will use huge_pte_offset() inside. Assert on the locks when walking the pgtable.
Note, the vma lock assertion will be a no-op for private mappings.
Document the last special case in the page_vma_mapped_walk() path where we don't need any more lock to call hugetlb_walk().
Taking vma lock there is not needed because either: (1) potential callers of hugetlb pvmw holds i_mmap_rwsem already (from one rmap_walk()), or (2) the caller will not walk a hugetlb vma at all so the hugetlb code path not reachable (e.g. in ksm or uprobe paths).
It's slightly implicit for future page_vma_mapped_walk() callers on that lock requirement. But anyway, when one day this rule breaks, one will get a straightforward warning in hugetlb_walk() with lockdep, then there'll be a way out.
--- fs/hugetlbfs/inode.c | 4 +--- fs/userfaultfd.c | 6 ++---- include/linux/hugetlb.h | 39 +++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 32 +++++++++++++------------------- mm/page_vma_mapped.c | 9 ++++++--- mm/pagewalk.c | 4 +--- 6 files changed, 62 insertions(+), 32 deletions(-)
fs/hugetlbfs/inode.c +1 -3
@@ -388,9 +388,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, {pte_t *ptep, pte; - ptep =huge_pte_offset (vma->vm_mm, addr, -huge_page_size (hstate_vma (vma))); - + ptep =hugetlb_walk (vma, addr,huge_page_size (hstate_vma (vma)));if (!ptep)return false;
fs/userfaultfd.c +2 -4
@@ -237,14 +237,12 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, unsigned long flags, unsigned long reason) { -struct mm_struct *mm = ctx->mm;pte_t *ptep, pte;bool ret = true; -mmap_assert_locked (mm); - - ptep =huge_pte_offset (mm, address,vma_mmu_pagesize (vma)); +mmap_assert_locked (ctx->mm); + ptep =hugetlb_walk (vma, address,vma_mmu_pagesize (vma));if (!ptep) goto out;
include/linux/hugetlb.h +39 -0
@@ -2,6 +2,7 @@#ifndef _LINUX_HUGETLB_H#define _LINUX_HUGETLB_H +#include <linux/mm.h> #include <linux/mm_types.h> #include <linux/mmdebug.h> #include <linux/fs.h>
@@ -196,6 +197,11 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,* huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. * Returns the pte_t* if found, or NULL if the address is not mapped. * +* IMPORTANT: we should normally not directly call this function, instead +* this is only a common interface to implement arch-specific +* walker. Please use hugetlb_walk() instead, because that will attempt to +* verify the locking for you. +* * Since this function will walk all the pgtable pages (including not only * high-level pgtable page, but also PUD entry that can be unshared * concurrently for VM_SHARED), the caller of this function should be
@@ -1229,4 +1235,37 @@ bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);#define flush_hugetlb_tlb_range (vma, addr, end) flush_tlb_range(vma, addr, end)#endif +static inline bool +__vma_shareable_flags_pmd (struct vm_area_struct *vma) +{ +return vma->vm_flags & (VM_MAYSHARE |VM_SHARED ) && + vma->vm_private_data; +} + +/* +* Safe version of huge_pte_offset() to check the locks. See comments +* above huge_pte_offset(). +*/ +static inline pte_t * +hugetlb_walk (struct vm_area_struct *vma,unsigned long addr,unsigned long sz) +{ +#if defined(CONFIG_HUGETLB_PAGE ) && \ + defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE ) && defined(CONFIG_LOCKDEP ) +struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; + +/* +* If pmd sharing possible, locking needed to safely walk the +* hugetlb pgtables. More information can be found at the comment +* above huge_pte_offset() in the same file. +* +* NOTE: lockdep_is_held() is only defined with CONFIG_LOCKDEP. +*/ +if (__vma_shareable_flags_pmd (vma)) +WARN_ON_ONCE (!lockdep_is_held (&vma_lock->rw_sema) && + !lockdep_is_held ( + &vma->vm_file->f_mapping->i_mmap_rwsem)); +#endif +return huge_pte_offset (vma->vm_mm, addr, sz); +} +#endif /* _LINUX_HUGETLB_H */
mm/hugetlb.c +13 -19
@@ -4814,7 +4814,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, }else {/* * For shared mappings the vma lock must be held before -* calling huge_pte_offset in the src vma. Otherwise, the +* calling hugetlb_walk() in the src vma. Otherwise, the * returned ptep could go away if part of a shared pmd and * another thread calls huge_pmd_unshare. */
@@ -4824,7 +4824,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask =hugetlb_mask_last_page (h);for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) {spinlock_t *src_ptl, *dst_ptl; - src_pte =huge_pte_offset (src, addr, sz); + src_pte =hugetlb_walk (src_vma, addr, sz);if (!src_pte) { addr |= last_addr_mask;continue ;
@@ -5028,7 +5028,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,hugetlb_vma_lock_write (vma);i_mmap_lock_write (mapping);for (; old_addr < old_end; old_addr += sz, new_addr += sz) { - src_pte =huge_pte_offset (mm, old_addr, sz); + src_pte =hugetlb_walk (vma, old_addr, sz);if (!src_pte) { old_addr |= last_addr_mask; new_addr |= last_addr_mask;
@@ -5091,7 +5091,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct last_addr_mask =hugetlb_mask_last_page (h); address = start;for (; address < end; address += sz) { - ptep =huge_pte_offset (mm, address, sz); + ptep =hugetlb_walk (vma, address, sz);if (!ptep) { address |= last_addr_mask;continue ;
@@ -5404,7 +5404,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,mutex_lock (&hugetlb_fault_mutex_table[hash]);hugetlb_vma_lock_read (vma);spin_lock (ptl); - ptep =huge_pte_offset (mm, haddr,huge_page_size (h)); + ptep =hugetlb_walk (vma, haddr,huge_page_size (h));if (likely (ptep &&pte_same (huge_ptep_get (ptep), pte))) goto retry_avoidcopy;
@@ -5442,7 +5442,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,* before the page tables are altered */ spin_lock (ptl); - ptep =huge_pte_offset (mm, haddr,huge_page_size (h)); + ptep =hugetlb_walk (vma, haddr,huge_page_size (h));if (likely (ptep &&pte_same (huge_ptep_get (ptep), pte))) {/* Break COW or unshare */ huge_ptep_clear_flush (vma, haddr, ptep);
@@ -6228,7 +6228,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,return NULL ;hugetlb_vma_lock_read (vma); - pte =huge_pte_offset (mm, haddr,huge_page_size (h)); + pte =hugetlb_walk (vma, haddr,huge_page_size (h));if (!pte) goto out_unlock;
@@ -6293,8 +6293,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,* * Note that page table lock is not held when pte is null. */ - pte =huge_pte_offset (mm, vaddr &huge_page_mask (h), -huge_page_size (h)); + pte =hugetlb_walk (vma, vaddr &huge_page_mask (h), +huge_page_size (h));if (pte) ptl =huge_pte_lock (h, mm, pte); absent = !pte ||huge_pte_none (huge_ptep_get (pte));
@@ -6480,7 +6480,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, last_addr_mask =hugetlb_mask_last_page (h);for (; address < end; address += psize) {spinlock_t *ptl; - ptep =huge_pte_offset (mm, address, psize); + ptep =hugetlb_walk (vma, address, psize);if (!ptep) { address |= last_addr_mask;continue ;
@@ -6858,12 +6858,6 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, *end =ALIGN (*end,PUD_SIZE ); } -static bool __vma_shareable_flags_pmd (struct vm_area_struct *vma) -{ -return vma->vm_flags & (VM_MAYSHARE |VM_SHARED ) && - vma->vm_private_data; -} -void hugetlb_vma_lock_read (struct vm_area_struct *vma) {if (__vma_shareable_flags_pmd (vma)) {
@@ -7029,8 +7023,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, saddr =page_table_shareable (svma, vma, addr, idx);if (saddr) { - spte =huge_pte_offset (svma->vm_mm, saddr, -vma_mmu_pagesize (svma)); + spte =hugetlb_walk (svma, saddr, +vma_mmu_pagesize (svma));if (spte) {get_page (virt_to_page (spte));break ;
@@ -7388,7 +7382,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)hugetlb_vma_lock_write (vma);i_mmap_lock_write (vma->vm_file->f_mapping);for (address = start; address < end; address +=PUD_SIZE ) { - ptep =huge_pte_offset (mm, address, sz); + ptep =hugetlb_walk (vma, address, sz);if (!ptep)continue ; ptl =huge_pte_lock (h, mm, ptep);
mm/page_vma_mapped.c +6 -3
@@ -168,9 +168,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)/* The only possible mapping was handled on last iteration */ if (pvmw->pte)return not_found (pvmw); - -/* when pud is not present, pte will be NULL */ - pvmw->pte =huge_pte_offset (mm, pvmw->address, size); +/* +* All callers that get here will already hold the +* i_mmap_rwsem. Therefore, no additional locks need to be +* taken before calling hugetlb_walk(). +*/ + pvmw->pte =hugetlb_walk (vma, pvmw->address, size);if (!pvmw->pte)return false;
mm/pagewalk.c +1 -3
@@ -305,13 +305,11 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,hugetlb_vma_lock_read (vma);do { next =hugetlb_entry_end (h, addr, end); - pte =huge_pte_offset (walk->mm, addr & hmask, sz); - + pte =hugetlb_walk (vma, addr & hmask, sz);if (pte) err = ops->hugetlb_entry (pte, hmask, addr, next, walk);else if (ops->pte_hole) err = ops->pte_hole (addr, next,-1 , walk); -if (err)break ; }while (addr = next, addr != end);
Signature · 1 line
2.37.3
Re: [PATCH v3 9/9] mm/hugetlb: Introduce hugetlb_walk()
Quoted 24 lines · Peter Xu
huge_pte_offset()is the main walker function for hugetlb pgtables. The name is not really representing what it does, though.Instead of renaming it, introduce a wrapper function called
hugetlb_walk()which will usehuge_pte_offset()inside. Assert on the locks when walking the pgtable.Note, the vma lock assertion will be a no-op for private mappings.
Document the last special case in the
page_vma_mapped_walk()path where we don't need any more lock to callhugetlb_walk().Taking vma lock there is not needed because either: (1) potential callers of hugetlb pvmw holds
i_mmap_rwsemalready (from onermap_walk()), or (2) the caller will not walk a hugetlb vma at all so the hugetlb code path not reachable (e.g. in ksm or uprobe paths).It's slightly implicit for future
page_vma_mapped_walk()callers on that lock requirement. But anyway, when one day this rule breaks, one will get a straightforward warning inhugetlb_walk()with lockdep, then there'll be a way out.Reviewed-by Mike Kravetz <[email protected]>Signed-off-by Peter Xu <[email protected]>
Quoted 9 lines
--- fs/hugetlbfs/inode.c | 4 +--- fs/userfaultfd.c | 6 ++---- include/linux/hugetlb.h | 39 +++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 32 +++++++++++++------------------- mm/page_vma_mapped.c | 9 ++++++--- mm/pagewalk.c | 4 +--- 6 files changed, 62 insertions(+), 32 deletions(-)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 257 lines
fs/hugetlbfs/inode.c +1
-3@@ -388,9 +388,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, {pte_t *ptep, pte; - ptep =huge_pte_offset (vma->vm_mm, addr, -huge_page_size (hstate_vma (vma))); - + ptep =hugetlb_walk (vma, addr,huge_page_size (hstate_vma (vma)));if (!ptep)return false;fs/userfaultfd.c +2
-4@@ -237,14 +237,12 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, unsigned long flags, unsigned long reason) { -struct mm_struct *mm = ctx->mm;pte_t *ptep, pte;bool ret = true; -mmap_assert_locked (mm); - - ptep =huge_pte_offset (mm, address,vma_mmu_pagesize (vma)); +mmap_assert_locked (ctx->mm); + ptep =hugetlb_walk (vma, address,vma_mmu_pagesize (vma));if (!ptep) goto out;include/linux/hugetlb.h +39
-0@@ -2,6 +2,7 @@#ifndef _LINUX_HUGETLB_H#define _LINUX_HUGETLB_H +#include <linux/mm.h> #include <linux/mm_types.h> #include <linux/mmdebug.h> #include <linux/fs.h> @@ -196,6 +197,11 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,* huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. * Returns the pte_t* if found, or NULL if the address is not mapped. * +* IMPORTANT: we should normally not directly call this function, instead +* this is only a common interface to implement arch-specific +* walker. Please use hugetlb_walk() instead, because that will attempt to +* verify the locking for you. +* * Since this function will walk all the pgtable pages (including not only * high-level pgtable page, but also PUD entry that can be unshared * concurrently for VM_SHARED), the caller of this function should be @@ -1229,4 +1235,37 @@ bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);#define flush_hugetlb_tlb_range (vma, addr, end) flush_tlb_range(vma, addr, end)#endif +static inline bool +__vma_shareable_flags_pmd (struct vm_area_struct *vma) +{ +return vma->vm_flags & (VM_MAYSHARE |VM_SHARED ) && + vma->vm_private_data; +} + +/* +* Safe version of huge_pte_offset() to check the locks. See comments +* above huge_pte_offset(). +*/ +static inline pte_t * +hugetlb_walk (struct vm_area_struct *vma,unsigned long addr,unsigned long sz) +{ +#if defined(CONFIG_HUGETLB_PAGE ) && \ + defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE ) && defined(CONFIG_LOCKDEP ) +struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; + +/* +* If pmd sharing possible, locking needed to safely walk the +* hugetlb pgtables. More information can be found at the comment +* above huge_pte_offset() in the same file. +* +* NOTE: lockdep_is_held() is only defined with CONFIG_LOCKDEP. +*/ +if (__vma_shareable_flags_pmd (vma)) +WARN_ON_ONCE (!lockdep_is_held (&vma_lock->rw_sema) && + !lockdep_is_held ( + &vma->vm_file->f_mapping->i_mmap_rwsem)); +#endif +return huge_pte_offset (vma->vm_mm, addr, sz); +} +#endif /* _LINUX_HUGETLB_H */ mm/hugetlb.c +13
-19@@ -4814,7 +4814,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, }else {/* * For shared mappings the vma lock must be held before -* calling huge_pte_offset in the src vma. Otherwise, the +* calling hugetlb_walk() in the src vma. Otherwise, the * returned ptep could go away if part of a shared pmd and * another thread calls huge_pmd_unshare. */ @@ -4824,7 +4824,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask =hugetlb_mask_last_page (h);for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) {spinlock_t *src_ptl, *dst_ptl; - src_pte =huge_pte_offset (src, addr, sz); + src_pte =hugetlb_walk (src_vma, addr, sz);if (!src_pte) { addr |= last_addr_mask;continue ;@@ -5028,7 +5028,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,hugetlb_vma_lock_write (vma);i_mmap_lock_write (mapping);for (; old_addr < old_end; old_addr += sz, new_addr += sz) { - src_pte =huge_pte_offset (mm, old_addr, sz); + src_pte =hugetlb_walk (vma, old_addr, sz);if (!src_pte) { old_addr |= last_addr_mask; new_addr |= last_addr_mask;@@ -5091,7 +5091,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct last_addr_mask =hugetlb_mask_last_page (h); address = start;for (; address < end; address += sz) { - ptep =huge_pte_offset (mm, address, sz); + ptep =hugetlb_walk (vma, address, sz);if (!ptep) { address |= last_addr_mask;continue ;@@ -5404,7 +5404,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,mutex_lock (&hugetlb_fault_mutex_table[hash]);hugetlb_vma_lock_read (vma);spin_lock (ptl); - ptep =huge_pte_offset (mm, haddr,huge_page_size (h)); + ptep =hugetlb_walk (vma, haddr,huge_page_size (h));if (likely (ptep &&pte_same (huge_ptep_get (ptep), pte))) goto retry_avoidcopy;@@ -5442,7 +5442,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,* before the page tables are altered */ spin_lock (ptl); - ptep =huge_pte_offset (mm, haddr,huge_page_size (h)); + ptep =hugetlb_walk (vma, haddr,huge_page_size (h));if (likely (ptep &&pte_same (huge_ptep_get (ptep), pte))) {/* Break COW or unshare */ huge_ptep_clear_flush (vma, haddr, ptep);@@ -6228,7 +6228,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,return NULL ;hugetlb_vma_lock_read (vma); - pte =huge_pte_offset (mm, haddr,huge_page_size (h)); + pte =hugetlb_walk (vma, haddr,huge_page_size (h));if (!pte) goto out_unlock;@@ -6293,8 +6293,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,* * Note that page table lock is not held when pte is null. */ - pte =huge_pte_offset (mm, vaddr &huge_page_mask (h), -huge_page_size (h)); + pte =hugetlb_walk (vma, vaddr &huge_page_mask (h), +huge_page_size (h));if (pte) ptl =huge_pte_lock (h, mm, pte); absent = !pte ||huge_pte_none (huge_ptep_get (pte));@@ -6480,7 +6480,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, last_addr_mask =hugetlb_mask_last_page (h);for (; address < end; address += psize) {spinlock_t *ptl; - ptep =huge_pte_offset (mm, address, psize); + ptep =hugetlb_walk (vma, address, psize);if (!ptep) { address |= last_addr_mask;continue ;@@ -6858,12 +6858,6 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, *end =ALIGN (*end,PUD_SIZE ); } -static bool __vma_shareable_flags_pmd (struct vm_area_struct *vma) -{ -return vma->vm_flags & (VM_MAYSHARE |VM_SHARED ) && - vma->vm_private_data; -} -void hugetlb_vma_lock_read (struct vm_area_struct *vma) {if (__vma_shareable_flags_pmd (vma)) {@@ -7029,8 +7023,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, saddr =page_table_shareable (svma, vma, addr, idx);if (saddr) { - spte =huge_pte_offset (svma->vm_mm, saddr, -vma_mmu_pagesize (svma)); + spte =hugetlb_walk (svma, saddr, +vma_mmu_pagesize (svma));if (spte) {get_page (virt_to_page (spte));break ;@@ -7388,7 +7382,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)hugetlb_vma_lock_write (vma);i_mmap_lock_write (vma->vm_file->f_mapping);for (address = start; address < end; address +=PUD_SIZE ) { - ptep =huge_pte_offset (mm, address, sz); + ptep =hugetlb_walk (vma, address, sz);if (!ptep)continue ; ptl =huge_pte_lock (h, mm, ptep);mm/page_vma_mapped.c +6
-3@@ -168,9 +168,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)/* The only possible mapping was handled on last iteration */ if (pvmw->pte)return not_found (pvmw); - -/* when pud is not present, pte will be NULL */ - pvmw->pte =huge_pte_offset (mm, pvmw->address, size); +/* +* All callers that get here will already hold the +* i_mmap_rwsem. Therefore, no additional locks need to be +* taken before calling hugetlb_walk(). +*/ + pvmw->pte =hugetlb_walk (vma, pvmw->address, size);if (!pvmw->pte)return false;mm/pagewalk.c +1
-3@@ -305,13 +305,11 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,hugetlb_vma_lock_read (vma);do { next =hugetlb_entry_end (h, addr, end); - pte =huge_pte_offset (walk->mm, addr & hmask, sz); - + pte =hugetlb_walk (vma, addr & hmask, sz);if (pte) err = ops->hugetlb_entry (pte, hmask, addr, next, walk);else if (ops->pte_hole) err = ops->pte_hole (addr, next,-1 , walk); -if (err)break ; }while (addr = next, addr != end);
Quoted 8 lines · Peter Xu
Based on latest mm-unstable (
dbccafc6e992).This can be seen as a follow-up series to Mike's recent hugetlb vma lock series for pmd unsharing, but majorly covering safe use of
huge_pte_offset.Changelog for
v2->v3:
- Added a-bs and r-bs
- Inwalk_hugetlb_range()hooks, don't release vma lock forcond_resched()
Thanks for your continued work on this Peter!
As a followup to that discussion about releasing vma lock for cond_resched(), I do not have a strong opinion. Glad you kept my r-b.
Signature · 1 line
Mike Kravetz