hugetlbfs: skip PMD unsharing when shareable lock unavailable
Subsystems (3)
PATCH v3 1/1 hugetlbfs: skip PMD unsharing when shareable lock unavailable
When hugetlb_vmdelete_list() cannot acquire the shareable lock for a VMA, the previous fix (dd83609b8898) skipped the entire VMA to avoid lock assertions in huge_pmd_unshare(). However, this prevented pages from being unmapped and freed, causing a regression in fallocate(PUNCH_HOLE) operations where pages were not freed immediately, as reported by Mark Brown.
The issue occurs because:
1. hugetlb_vmdelete_list() calls hugetlb_vma_trylock_write()
2. For shareable VMAs, this attempts to acquire the shareable lock
3. If successful, huge_pmd_unshare() expects the lock to be held
4. huge_pmd_unshare() asserts the lock via hugetlb_vma_assert_locked()
The v2 fix avoided calling code that requires locks, but this prevented page unmapping entirely, breaking the expected behavior where pages are freed during punch hole operations.
This v3 fix takes a different approach: instead of skipping the entire VMA, we skip only the PMD unsharing operation when we don't have the required lock, while still proceeding with page unmapping. This is safe because:
- PMD unsharing is an optimization to reduce shared page table overhead
- Page unmapping can proceed safely with just the VMA write lock
- Pages get freed immediately as expected by PUNCH_HOLE operations
- The PMD metadata will be cleaned up when the VMA is destroyed
We introduce a new ZAP_FLAG_NO_UNSHARE flag that communicates to __unmap_hugepage_range() that it should skip huge_pmd_unshare() while still clearing page table entries and freeing pages.
---
Changes in v3:
- Instead of skipping entire VMAs, skip only PMD unsharing operation
- Add ZAP_FLAG_NO_UNSHARE flag to communicate lock status
- Ensure pages are still unmapped and freed immediately
- Fixes regression in fallocate PUNCH_HOLE reported by Mark Brown
Changes in v2:
- Check for shareable lock before trylock to avoid lock leaks
- Add comment explaining why non-shareable VMAs are skipped
--- fs/hugetlbfs/inode.c | 22 ++++++++++++---------- include/linux/mm.h | 2 ++ mm/hugetlb.c | 3 ++- 3 files changed, 16 insertions(+), 11 deletions(-)
fs/hugetlbfs/inode.c +12 -10
@@ -474,29 +474,31 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) {unsigned long v_start;unsigned long v_end; +bool have_shareable_lock; +zap_flags_t local_flags = zap_flags;if (!hugetlb_vma_trylock_write (vma))continue ; - + + have_shareable_lock =__vma_shareable_lock (vma); +/* -* Skip VMAs without shareable locks. Per the design in commit -* 40549ba8f8e0, these will be handled by remove_inode_hugepages() -* called after this function with proper locking. +* If we can't get the shareable lock, set ZAP_FLAG_NO_UNSHARE +* to skip PMD unsharing. We still proceed with unmapping to +* ensure pages are properly freed, which is critical for punch +* hole operations that expect immediate page freeing. */ -if (!__vma_shareable_lock (vma)) - goto skip; - +if (!have_shareable_lock) + local_flags |=ZAP_FLAG_NO_UNSHARE ; v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -unmap_hugepage_range (vma, v_start, v_end,NULL , zap_flags); - +unmap_hugepage_range (vma, v_start, v_end,NULL , local_flags);/* * Note that vma lock only exists for shared/non-private * vmas. Therefore, lock is not held when calling * unmap_hugepage_range for private vmas. */ -skip:hugetlb_vma_unlock_write (vma); } }
include/linux/mm.h +2 -0
@@ -2395,6 +2395,8 @@ struct zap_details {#define ZAP_FLAG_DROP_MARKER ((__force zap_flags_t) BIT(0))/* Set in unmap_vmas() to indicate a final unmap call. Only used by hugetlb */ #define ZAP_FLAG_UNMAP ((__force zap_flags_t) BIT(1)) +/* Skip PMD unsharing when unmapping hugetlb ranges without shareable lock */ +#define ZAP_FLAG_NO_UNSHARE ((__force zap_flags_t) BIT(2))#ifdef CONFIG_SCHED_MM_CID void sched_mm_cid_before_execve (struct task_struct *t);
mm/hugetlb.c +2 -1
@@ -5885,7 +5885,8 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, } ptl =huge_pte_lock (h, mm, ptep); -if (huge_pmd_unshare (mm, vma, address, ptep)) { +if (!(zap_flags &ZAP_FLAG_NO_UNSHARE ) && +huge_pmd_unshare (mm, vma, address, ptep)) {spin_unlock (ptl);tlb_flush_pmd_range (tlb, address &PUD_MASK ,PUD_SIZE ); force_flush = true;
Signature · 1 line
2.43.0
Quoted 2 lines · Deepanshu Kartikey
When
hugetlb_vmdelete_list()cannot acquire the shareable lock for a VMA, the previous fix (dd83609b8898) skipped the entire VMA to avoid lock
The proper way to mention a commit here
"... fix in commit dd83609b8898 ("hugetlbfs: skip VMAs without shareable locks in hugetlb_vmdelete_list") skipped ..."
Quoted 10 lines
assertions in
huge_pmd_unshare(). However, this prevented pages from being unmapped and freed, causing a regression infallocate(PUNCH_HOLE)operations where pages were not freed immediately, as reported by Mark Brown.The issue occurs because:
1.hugetlb_vmdelete_list()callshugetlb_vma_trylock_write()
2. For shareable VMAs, this attempts to acquire the shareable lock
3. If successful,huge_pmd_unshare()expects the lock to be held
4.huge_pmd_unshare()asserts the lock viahugetlb_vma_assert_locked()
Quoted 7 lines
The v2 fix avoided calling code that requires locks, but this prevented page unmapping entirely, breaking the expected behavior where pages are freed during punch hole operations.
This v3 fix takes a different approach: instead of skipping the entire VMA, we skip only the PMD unsharing operation when we don't have the required lock, while still proceeding with page unmapping. This is safe because:
It's confusing to talk about fix versions. If you want to reference previous discussions, rather link to them.
Quoted 44 lines
- PMD unsharing is an optimization to reduce shared page table overhead
- Page unmapping can proceed safely with just the VMA write lock
- Pages get freed immediately as expected byPUNCH_HOLEoperations
- The PMD metadata will be cleaned up when the VMA is destroyedWe introduce a new
ZAP_FLAG_NO_UNSHAREflag that communicates to__unmap_hugepage_range()that it should skiphuge_pmd_unshare()while still clearing page table entries and freeing pages.Reported-by [email protected]Reported-by Mark Brown <[email protected]>Fixes dd83609b8898 ("hugetlbfs: skip VMAs without shareable locks in hugetlb_vmdelete_list")Tested-by [email protected]Signed-off-by Deepanshu Kartikey <[email protected]>---
Changes in v3:
- Instead of skipping entire VMAs, skip only PMD unsharing operation
- AddZAP_FLAG_NO_UNSHAREflag to communicate lock status
- Ensure pages are still unmapped and freed immediately
- Fixes regression in fallocatePUNCH_HOLEreported by Mark BrownChanges in v2:
- Check for shareable lock before trylock to avoid lock leaks
- Add comment explaining why non-shareable VMAs are skipped--- fs/hugetlbfs/inode.c | 22 ++++++++++++---------- include/linux/mm.h | 2 ++ mm/hugetlb.c | 3 ++- 3 files changed, 16 insertions(+), 11 deletions(-)fs/hugetlbfs/inode.c +2
-0@@ -474,29 +474,31 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) {unsigned long v_start;unsigned long v_end; +bool have_shareable_lock; +zap_flags_t local_flags = zap_flags;if (!hugetlb_vma_trylock_write (vma))
fs/hugetlbfs/inode.c hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, · line 486
Quoted 10 lines
fs/hugetlbfs/inode.c +4
-4continue ; - + + have_shareable_lock =__vma_shareable_lock (vma); +/* -* Skip VMAs without shareable locks. Per the design in commit -* 40549ba8f8e0, these will be handled by remove_inode_hugepages() -* called after this function with proper locking. +* If we can't get the shareable lock, set ZAP_FLAG_NO_UNSHARE
What do you mean with "If we can't get the shareable lock"? __vma_shareable_lock() doesn't tell us whether we grabbed the lock, but whether we have to grab the lock?
I see now R-b/Ack from hugetlb maintainers and this seems to be getting rather complicated now and I cannot really easily judge what's right or wrong now.
@Muchun, Oscar, can you take a look?
Quoted 29 lines
fs/hugetlbfs/inode.c +6
-6+* to skip PMD unsharing. We still proceed with unmapping to +* ensure pages are properly freed, which is critical for punch +* hole operations that expect immediate page freeing. */ -if (!__vma_shareable_lock (vma)) - goto skip; - +if (!have_shareable_lock) + local_flags |=ZAP_FLAG_NO_UNSHARE ; v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -unmap_hugepage_range (vma, v_start, v_end,NULL , zap_flags); - +unmap_hugepage_range (vma, v_start, v_end,NULL , local_flags);/* * Note that vma lock only exists for shared/non-private * vmas. Therefore, lock is not held when calling * unmap_hugepage_range for private vmas. */ -skip:hugetlb_vma_unlock_write (vma); } }include/linux/mm.h +0
-0
include/linux/mm.h struct zap_details { · line 2399
Quoted 5 lines
include/linux/mm.h +2
-0@@ -2395,6 +2395,8 @@ struct zap_details {#define ZAP_FLAG_DROP_MARKER ((__force zap_flags_t) BIT(0))/* Set in unmap_vmas() to indicate a final unmap call. Only used by hugetlb */ #define ZAP_FLAG_UNMAP ((__force zap_flags_t) BIT(1)) +/* Skip PMD unsharing when unmapping hugetlb ranges without shareable lock */ +#define ZAP_FLAG_NO_UNSHARE ((__force zap_flags_t) BIT(2))
That's nasty: this is hugetlb-specific stuff in a generic mm.h header using generic mm flags.
I'm sure we can find a way communicate that in a different way within hugetlb code and leave the generic ZAP_* flags alone?
Signature · 3 lines
Cheers David / dhildenb
Oscar Salvador
maintainerQuoted 36 lines · Deepanshu Kartikey
When
hugetlb_vmdelete_list()cannot acquire the shareable lock for a VMA, the previous fix (dd83609b8898) skipped the entire VMA to avoid lock assertions inhuge_pmd_unshare(). However, this prevented pages from being unmapped and freed, causing a regression infallocate(PUNCH_HOLE)operations where pages were not freed immediately, as reported by Mark Brown.The issue occurs because:
1.hugetlb_vmdelete_list()callshugetlb_vma_trylock_write()
2. For shareable VMAs, this attempts to acquire the shareable lock
3. If successful,huge_pmd_unshare()expects the lock to be held
4.huge_pmd_unshare()asserts the lock viahugetlb_vma_assert_locked()The v2 fix avoided calling code that requires locks, but this prevented page unmapping entirely, breaking the expected behavior where pages are freed during punch hole operations.
This v3 fix takes a different approach: instead of skipping the entire VMA, we skip only the PMD unsharing operation when we don't have the required lock, while still proceeding with page unmapping. This is safe because:
- PMD unsharing is an optimization to reduce shared page table overhead
- Page unmapping can proceed safely with just the VMA write lock
- Pages get freed immediately as expected byPUNCH_HOLEoperations
- The PMD metadata will be cleaned up when the VMA is destroyedWe introduce a new
ZAP_FLAG_NO_UNSHAREflag that communicates to__unmap_hugepage_range()that it should skiphuge_pmd_unshare()while still clearing page table entries and freeing pages.Reported-by [email protected]Reported-by Mark Brown <[email protected]>Fixes dd83609b8898 ("hugetlbfs: skip VMAs without shareable locks in hugetlb_vmdelete_list")Tested-by [email protected]Signed-off-by Deepanshu Kartikey <[email protected]>---
...
Quoted 21 lines
fs/hugetlbfs/inode.c +5
-4@@ -474,29 +474,31 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) {unsigned long v_start;unsigned long v_end; +bool have_shareable_lock; +zap_flags_t local_flags = zap_flags;if (!hugetlb_vma_trylock_write (vma))continue ; - + + have_shareable_lock =__vma_shareable_lock (vma); +/* -* Skip VMAs without shareable locks. Per the design in commit -* 40549ba8f8e0, these will be handled by remove_inode_hugepages() -* called after this function with proper locking.
fs/hugetlbfs/inode.c hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, · line 492
Quoted 10 lines
fs/hugetlbfs/inode.c +6
-3+* If we can't get the shareable lock, set ZAP_FLAG_NO_UNSHARE +* to skip PMD unsharing. We still proceed with unmapping to +* ensure pages are properly freed, which is critical for punch +* hole operations that expect immediate page freeing. */ -if (!__vma_shareable_lock (vma)) - goto skip; - +if (!have_shareable_lock) + local_flags |=ZAP_FLAG_NO_UNSHARE ;
This is quite a head-spinning thing.
First of all, as David pointed out, that comment is misleading as it looks like __vma_shareable_lock() performs a taking action which is not true, so that should reworded.
Now, the thing is:
- Prior to commit dd83609b8898("hugetlbfs: skip VMAs without shareable locks in hugetlb_vmdelete_list"), we were unconditionally calling huge_pmd_unshare(), which asserted the vma lock and we didn't hold it. My question would be that Mike's vma-lock addition happened in 2022, how's that we didn't see this sooner? It should be rather easy to trigger? I'm a bit puzzled.
- Ok, since there's nothing to unshare, we skip the vma here and remove_inode_hugepages() should take care of it.
But that seems to be troublesome because on punch-hole operation pages don't get freed.
- So instead, we just skip the unsharing operation and keep carrying with the unmapping/freeing in __unmap_hugepage_range.
I don't know but to me it seems that we're going to large extends to fix an assertion.
So, the thing is, can't we check __vma_shareable_lock in __unmap_hugepage_range() and only call huge_pmd_unshare() if we need to?
Signature · 2 lines
Oscar Salvador SUSE Labs
PATCH v4 1/1 hugetlbfs: check for shareable lock before calling huge_pmd_unshare()
In reply to Deepanshu KartikeyWhen hugetlb_vmdelete_list() processes VMAs during truncate operations, it may encounter VMAs where huge_pmd_unshare() is called without the required shareable lock. This triggers an assertion failure in hugetlb_vma_assert_locked().
The previous fix in commit dd83609b8898 ("hugetlbfs: skip VMAs without shareable locks in hugetlb_vmdelete_list") skipped entire VMAs without shareable locks to avoid the assertion. However, this prevented pages from being unmapped and freed, causing a regression in fallocate(PUNCH_HOLE) operations where pages were not freed immediately, as reported by Mark Brown.
Instead of skipping VMAs or adding new flags, check __vma_shareable_lock() directly in __unmap_hugepage_range() right before calling huge_pmd_unshare(). This ensures PMD unsharing only happens when the VMA has a shareable lock structure, while still allowing page unmapping and freeing to proceed for all VMAs.
---
Changes in v4:
- Simplified approach per Oscar's suggestion: check __vma_shareable_lock() directly in __unmap_hugepage_range() before calling huge_pmd_unshare()
- Removed ZAP_FLAG_NO_UNSHARE flag per David's feedback to avoid polluting generic mm.h header
- Reverted hugetlb_vmdelete_list() to not skip VMAs
Changes in v3:
- Added ZAP_FLAG_NO_UNSHARE to skip only PMD unsharing, not entire VMA
Changes in v2:
- Skip entire VMAs without shareable locks in hugetlb_vmdelete_list() (caused PUNCH_HOLE regression)
Changes in v1:
- Initial fix attempt
--- fs/hugetlbfs/inode.c | 10 +--------- mm/hugetlb.c | 2 +- 2 files changed, 2 insertions(+), 10 deletions(-)
fs/hugetlbfs/inode.c +1 -9
@@ -478,14 +478,6 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,if (!hugetlb_vma_trylock_write (vma))continue ; -/* -* Skip VMAs without shareable locks. Per the design in commit -* 40549ba8f8e0, these will be handled by remove_inode_hugepages() -* called after this function with proper locking. -*/ -if (!__vma_shareable_lock (vma)) - goto skip; - v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end);
@@ -496,7 +488,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,* vmas. Therefore, lock is not held when calling * unmap_hugepage_range for private vmas. */ -skip: +hugetlb_vma_unlock_write (vma); } }
mm/hugetlb.c +1 -1
@@ -5885,7 +5885,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, } ptl =huge_pte_lock (h, mm, ptep); -if (huge_pmd_unshare (mm, vma, address, ptep)) { +if (__vma_shareable_lock (vma) &&huge_pmd_unshare (mm, vma, address, ptep)) {spin_unlock (ptl);tlb_flush_pmd_range (tlb, address &PUD_MASK ,PUD_SIZE ); force_flush = true;
Signature · 1 line
2.43.0
Re: [PATCH v4] hugetlbfs: check for shareable lock before calling huge_pmd_unshare()
Quoted 17 lines · Deepanshu Kartikey
When
hugetlb_vmdelete_list()processes VMAs during truncate operations, it may encounter VMAs wherehuge_pmd_unshare()is called without the required shareable lock. This triggers an assertion failure inhugetlb_vma_assert_locked().The previous fix in commit
dd83609b8898("hugetlbfs: skip VMAs without shareable locks inhugetlb_vmdelete_list") skipped entire VMAs without shareable locks to avoid the assertion. However, this prevented pages from being unmapped and freed, causing a regression infallocate(PUNCH_HOLE)operations where pages were not freed immediately, as reported by Mark Brown.Instead of skipping VMAs or adding new flags, check
__vma_shareable_lock()directly in__unmap_hugepage_range()right before callinghuge_pmd_unshare(). This ensures PMD unsharing only happens when the VMA has a shareable lock structure, while still allowing page unmapping and freeing to proceed for all VMAs.
Quoted 10 lines
Reported-by [email protected]Tested-by [email protected]Reported-by Mark Brown <[email protected]>Fixes dd83609b8898 ("hugetlbfs: skip VMAs without shareable locks in hugetlb_vmdelete_list")Suggested-by Oscar Salvador <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Deepanshu Kartikey <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v4] hugetlbfs: check for shareable lock before calling huge_pmd_unshare()
In reply to Deepanshu KartikeyQuoted 82 lines · Deepanshu Kartikey
When
hugetlb_vmdelete_list()processes VMAs during truncate operations, it may encounter VMAs wherehuge_pmd_unshare()is called without the required shareable lock. This triggers an assertion failure inhugetlb_vma_assert_locked().The previous fix in commit
dd83609b8898("hugetlbfs: skip VMAs without shareable locks inhugetlb_vmdelete_list") skipped entire VMAs without shareable locks to avoid the assertion. However, this prevented pages from being unmapped and freed, causing a regression infallocate(PUNCH_HOLE)operations where pages were not freed immediately, as reported by Mark Brown.Instead of skipping VMAs or adding new flags, check
__vma_shareable_lock()directly in__unmap_hugepage_range()right before callinghuge_pmd_unshare(). This ensures PMD unsharing only happens when the VMA has a shareable lock structure, while still allowing page unmapping and freeing to proceed for all VMAs.Reported-by [email protected]Tested-by [email protected]Reported-by Mark Brown <[email protected]>Fixes dd83609b8898 ("hugetlbfs: skip VMAs without shareable locks in hugetlb_vmdelete_list")Suggested-by Oscar Salvador <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Deepanshu Kartikey <[email protected]>---
Changes in v4:
- Simplified approach per Oscar's suggestion: check__vma_shareable_lock()directly in__unmap_hugepage_range()before callinghuge_pmd_unshare()
- RemovedZAP_FLAG_NO_UNSHAREflag per David's feedback to avoid polluting genericmm.hheader
- Revertedhugetlb_vmdelete_list()to not skip VMAsChanges in v3:
- AddedZAP_FLAG_NO_UNSHAREto skip only PMD unsharing, not entire VMAChanges in v2:
- Skip entire VMAs without shareable locks inhugetlb_vmdelete_list()(causedPUNCH_HOLEregression)Changes in v1:
- Initial fix attempt--- fs/hugetlbfs/inode.c | 10 +--------- mm/hugetlb.c | 2 +- 2 files changed, 2 insertions(+), 10 deletions(-)fs/hugetlbfs/inode.c +1
-9@@ -478,14 +478,6 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,if (!hugetlb_vma_trylock_write (vma))continue ; -/* -* Skip VMAs without shareable locks. Per the design in commit -* 40549ba8f8e0, these will be handled by remove_inode_hugepages() -* called after this function with proper locking. -*/ -if (!__vma_shareable_lock (vma)) - goto skip; - v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end);@@ -496,7 +488,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,* vmas. Therefore, lock is not held when calling * unmap_hugepage_range for private vmas. */ -skip: +hugetlb_vma_unlock_write (vma); } }mm/hugetlb.c +0
-0
mm/hugetlb.c void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 5891
Quoted 8 lines
mm/hugetlb.c +1
-1@@ -5885,7 +5885,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, } ptl =huge_pte_lock (h, mm, ptep); -if (huge_pmd_unshare (mm, vma, address, ptep)) { +if (__vma_shareable_lock (vma) &&huge_pmd_unshare (mm, vma, address, ptep)) {spin_unlock (ptl);tlb_flush_pmd_range (tlb, address &PUD_MASK ,PUD_SIZE ); force_flush = true;
Wondering, couldn't we handle that in huge_pmd_unshare()?
mm/hugetlb.c +3 -2
@@ -7598,13 +7598,14 @@ int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,p4d_t *p4d =p4d_offset (pgd, addr);pud_t *pud =pud_offset (p4d, addr); -i_mmap_assert_write_locked (vma->vm_file->f_mapping); -hugetlb_vma_assert_locked (vma);if (sz !=PMD_SIZE )return 0 ;if (!ptdesc_pmd_pts_count (virt_to_ptdesc (ptep)))return 0 ; +i_mmap_assert_write_locked (vma->vm_file->f_mapping); +hugetlb_vma_assert_locked (vma); +pud_clear (pud);/* * Once our caller drops the rmap lock, some other process might be
Signature · 3 lines
Cheers David / dhildenb