mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure
Subsystems (2)
PATCH v2 1/1 mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure
From: Ye Liu <[email protected]>
__khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the corresponding mm_slot. If mm_slot_alloc() fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.
This leaves the mm in an inconsistent state: it is marked as registered (MMF_VM_HUGEPAGE set), but will never be scanned by khugepaged. Future attempts to register the mm are skipped since khugepaged_enter_vma() checks the flag and returns early.
Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails, restoring the ability to retry registration later.
---
Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yang
mm/khugepaged.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/khugepaged.c +3 -1
@@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)return ; slot =mm_slot_alloc (mm_slot_cache); -if (!slot) +if (!slot) { +mm_flags_clear (MMF_VM_HUGEPAGE , mm);return ; + }spin_lock (&khugepaged_mm_lock);mm_slot_insert (mm_slots_hash, mm, slot);
Signature · 1 line
2.43.0
Quoted 12 lines · Ye Liu
From: Ye Liu <[email protected]>
__khugepaged_enter()setsMMF_VM_HUGEPAGEbefore allocating the correspondingmm_slot. Ifmm_slot_alloc()fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.This leaves the mm in an inconsistent state: it is marked as registered (
MMF_VM_HUGEPAGEset), but will never be scanned by khugepaged. Future attempts to register the mm are skipped sincekhugepaged_enter_vma()checks the flag and returns early.
Quoted 4 lines
Fix this by clearing
MMF_VM_HUGEPAGEwhenmm_slot_alloc()fails, restoring the ability to retry registration later.Fixes 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")
Nit: the title does not exactly match the original commit subject.
It should be
which includes quotes around "struct mm_slot".
No need to resend; I think Andrew can fix it up when applying :)
Quoted 2 lines
Signed-off-by Ye Liu <[email protected]>---
Thanks, LGTM.
Quoted 12 lines · Ye Liu
From: Ye Liu <[email protected]>
__khugepaged_enter()setsMMF_VM_HUGEPAGEbefore allocating the correspondingmm_slot. Ifmm_slot_alloc()fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.This leaves the mm in an inconsistent state: it is marked as registered (
MMF_VM_HUGEPAGEset), but will never be scanned by khugepaged. Future attempts to register the mm are skipped sincekhugepaged_enter_vma()checks the flag and returns early.
Quoted 6 lines
Fix this by clearing
MMF_VM_HUGEPAGEwhenmm_slot_alloc()fails, restoring the ability to retry registration later.Fixes 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")Signed-off-by Ye Liu <[email protected]>---
Good catch. Feel free to add:
Quoted 29 lines · Ye Liu
From: Ye Liu <[email protected]>
__khugepaged_enter()setsMMF_VM_HUGEPAGEbefore allocating the correspondingmm_slot. Ifmm_slot_alloc()fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.This leaves the mm in an inconsistent state: it is marked as registered (
MMF_VM_HUGEPAGEset), but will never be scanned by khugepaged. Future attempts to register the mm are skipped sincekhugepaged_enter_vma()checks the flag and returns early.Fix this by clearing
MMF_VM_HUGEPAGEwhenmm_slot_alloc()fails, restoring the ability to retry registration later.Fixes 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")Signed-off-by Ye Liu <[email protected]>---
Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yangmm/khugepaged.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/khugepaged.c +0
-0
mm/khugepaged.c void __khugepaged_enter(struct mm_struct *mm) · line 565
Quoted 8 lines
mm/khugepaged.c +3
-1@@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)return ; slot =mm_slot_alloc (mm_slot_cache); -if (!slot) +if (!slot) { +mm_flags_clear (MMF_VM_HUGEPAGE , mm);return ; + }
Note that, a racing khugepaged_enter_vma() may back off when it sees that MMF_VM_HUGEPAGE is set, but then the above clears the flag after slot alloc failure. So we end up not registering the mm with khugepaged. But I am sure no one cares, we are in much big trouble if slot alloc is failing.
Although one could argue the same about this patch, I will still say it is important to fix "flag is set but not registered with khugepaged" because that just feels wrong.
Quoted 3 lines
mm/khugepaged.c +0
-0spin_lock (&khugepaged_mm_lock);mm_slot_insert (mm_slots_hash, mm, slot);
Lance Yang
reviewerQuoted 34 lines · Dev Jain
Quoted 31 lines · Ye Liu
From: Ye Liu <[email protected]>
__khugepaged_enter()setsMMF_VM_HUGEPAGEbefore allocating the correspondingmm_slot. Ifmm_slot_alloc()fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.This leaves the mm in an inconsistent state: it is marked as registered (
MMF_VM_HUGEPAGEset), but will never be scanned by khugepaged. Future attempts to register the mm are skipped sincekhugepaged_enter_vma()checks the flag and returns early.Fix this by clearing
MMF_VM_HUGEPAGEwhenmm_slot_alloc()fails, restoring the ability to retry registration later.Fixes 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")Signed-off-by Ye Liu <[email protected]>---
Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yangmm/khugepaged.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/khugepaged.c +0
-0@@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)return ;
Quoted 12 lines
mm/khugepaged.c void __khugepaged_enter(struct mm_struct *mm) · line 565
Quoted 6 lines
mm/khugepaged.c +3
-1slot =mm_slot_alloc (mm_slot_cache); -if (!slot) +if (!slot) { +mm_flags_clear (MMF_VM_HUGEPAGE , mm);return ; + }Note that, a racing
khugepaged_enter_vma()may back off when it sees thatMMF_VM_HUGEPAGEis set, but then the above clears the flag after slot alloc failure. So we end up not registering the mm with khugepaged. But I am sure no one cares, we are in much big trouble if slot alloc is failing.
Right. A racing khugepaged_enter_vma() can see MMF_VM_HUGEPAGE is set and return, then !slot clears it again. If there is no later khugepaged_enter_vma(), the mm still wouldn't get registered :)
Quoted 3 lines
Although one could argue the same about this patch, I will still say it is important to fix "flag is set but not registered with khugepaged" because that just feels wrong.
+1 that is still better then keeping MMF_VM_HUGEPAGE set forever: any later attempt can retry and install the mm_slot :)
Cheers, Lance
David Hildenbrand (Arm)
maintainerQuoted 49 lines · Lance Yang
Quoted 34 lines · Dev Jain
Quoted 31 lines · Ye Liu
From: Ye Liu <[email protected]>
__khugepaged_enter()setsMMF_VM_HUGEPAGEbefore allocating the correspondingmm_slot. Ifmm_slot_alloc()fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.This leaves the mm in an inconsistent state: it is marked as registered (
MMF_VM_HUGEPAGEset), but will never be scanned by khugepaged. Future attempts to register the mm are skipped sincekhugepaged_enter_vma()checks the flag and returns early.Fix this by clearing
MMF_VM_HUGEPAGEwhenmm_slot_alloc()fails, restoring the ability to retry registration later.Fixes 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")Signed-off-by Ye Liu <[email protected]>---
Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yangmm/khugepaged.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/khugepaged.c +0
-0@@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)return ;Quoted 12 lines
mm/khugepaged.c void __khugepaged_enter(struct mm_struct *mm) · line 565
Quoted 6 lines
mm/khugepaged.c +3
-1slot =mm_slot_alloc (mm_slot_cache); -if (!slot) +if (!slot) { +mm_flags_clear (MMF_VM_HUGEPAGE , mm);return ; + }Note that, a racing
khugepaged_enter_vma()may back off when it sees thatMMF_VM_HUGEPAGEis set, but then the above clears the flag after slot alloc failure. So we end up not registering the mm with khugepaged. But I am sure no one cares, we are in much big trouble if slot alloc is failing.
Quoted 3 lines
Right. A racing
khugepaged_enter_vma()can seeMMF_VM_HUGEPAGEis set and return, then !slot clears it again. If there is no laterkhugepaged_enter_vma(), the mm still wouldn't get registered :)
So why not
mm/khugepaged.c +5 -2
@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)/* __khugepaged_exit() must not run from under us */ VM_BUG_ON_MM (collapse_test_exit (mm), mm); -if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) -return ; slot =mm_slot_alloc (mm_slot_cache);if (!slot)return ; +if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) { +mm_slot_free (mm_slot_cache, slot); +return ; + } +spin_lock (&khugepaged_mm_lock);mm_slot_insert (mm_slots_hash, mm, slot);/*
Arguably, on the race described above, likely the thread seeing the MMF_VM_HUGEPAGE would likely similarly have failed the allocation.
I'm fine with either, just wanted to raise the (cleaner looking?) alternative where we just properly back off?
Signature · 3 lines
Cheers, David
Lance Yang
reviewerQuoted 81 lines · David Hildenbrand (Arm)
Quoted 49 lines · Lance Yang
Quoted 34 lines · Dev Jain
Quoted 31 lines · Ye Liu
From: Ye Liu <[email protected]>
__khugepaged_enter()setsMMF_VM_HUGEPAGEbefore allocating the correspondingmm_slot. Ifmm_slot_alloc()fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.This leaves the mm in an inconsistent state: it is marked as registered (
MMF_VM_HUGEPAGEset), but will never be scanned by khugepaged. Future attempts to register the mm are skipped sincekhugepaged_enter_vma()checks the flag and returns early.Fix this by clearing
MMF_VM_HUGEPAGEwhenmm_slot_alloc()fails, restoring the ability to retry registration later.Fixes 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")Signed-off-by Ye Liu <[email protected]>---
Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yangmm/khugepaged.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/khugepaged.c +0
-0@@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)return ;Quoted 12 lines
mm/khugepaged.c void __khugepaged_enter(struct mm_struct *mm) · line 448
Quoted 6 lines
mm/khugepaged.c +3
-1slot =mm_slot_alloc (mm_slot_cache); -if (!slot) +if (!slot) { +mm_flags_clear (MMF_VM_HUGEPAGE , mm);return ; + }Note that, a racing
khugepaged_enter_vma()may back off when it sees thatMMF_VM_HUGEPAGEis set, but then the above clears the flag after slot alloc failure. So we end up not registering the mm with khugepaged. But I am sure no one cares, we are in much big trouble if slot alloc is failing.Quoted 3 lines
Right. A racing
khugepaged_enter_vma()can seeMMF_VM_HUGEPAGEis set and return, then !slot clears it again. If there is no laterkhugepaged_enter_vma(), the mm still wouldn't get registered :)So why not
mm/khugepaged.c +5
-2@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)/* __khugepaged_exit() must not run from under us */ VM_BUG_ON_MM (collapse_test_exit (mm), mm); -if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) -return ; slot =mm_slot_alloc (mm_slot_cache);if (!slot)return ; +if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) { +mm_slot_free (mm_slot_cache, slot); +return ; + } +spin_lock (&khugepaged_mm_lock);mm_slot_insert (mm_slots_hash, mm, slot);/*
Quoted 2 lines
Arguably, on the race described above, likely the thread seeing the
MMF_VM_HUGEPAGEwould likely similarly have failed the allocation.
Right, LGTM!
Quoted 2 lines
I'm fine with either, just wanted to raise the (cleaner looking?) alternative where we just properly back off?
Dev suggested the same thing[1] on v1 as well. We should have gone that way :)
Allocating the slot first and only setting MMF_VM_HUGEPAGE after that makes the race go away. If mm_slot_alloc() fails, there is nothing to undo.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Cheers, Lance
Dev Jain
reviewerQuoted 81 lines · David Hildenbrand (Arm)
Quoted 49 lines · Lance Yang
Quoted 34 lines · Dev Jain
Quoted 31 lines · Ye Liu
From: Ye Liu <[email protected]>
__khugepaged_enter()setsMMF_VM_HUGEPAGEbefore allocating the correspondingmm_slot. Ifmm_slot_alloc()fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures.This leaves the mm in an inconsistent state: it is marked as registered (
MMF_VM_HUGEPAGEset), but will never be scanned by khugepaged. Future attempts to register the mm are skipped sincekhugepaged_enter_vma()checks the flag and returns early.Fix this by clearing
MMF_VM_HUGEPAGEwhenmm_slot_alloc()fails, restoring the ability to retry registration later.Fixes 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")Signed-off-by Ye Liu <[email protected]>---
Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yangmm/khugepaged.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/khugepaged.c +0
-0@@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)return ;Quoted 12 lines
mm/khugepaged.c void __khugepaged_enter(struct mm_struct *mm) · line 448
Quoted 6 lines
mm/khugepaged.c +3
-1slot =mm_slot_alloc (mm_slot_cache); -if (!slot) +if (!slot) { +mm_flags_clear (MMF_VM_HUGEPAGE , mm);return ; + }Note that, a racing
khugepaged_enter_vma()may back off when it sees thatMMF_VM_HUGEPAGEis set, but then the above clears the flag after slot alloc failure. So we end up not registering the mm with khugepaged. But I am sure no one cares, we are in much big trouble if slot alloc is failing.Quoted 3 lines
Right. A racing
khugepaged_enter_vma()can seeMMF_VM_HUGEPAGEis set and return, then !slot clears it again. If there is no laterkhugepaged_enter_vma(), the mm still wouldn't get registered :)So why not
mm/khugepaged.c +5
-2@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)/* __khugepaged_exit() must not run from under us */ VM_BUG_ON_MM (collapse_test_exit (mm), mm); -if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) -return ; slot =mm_slot_alloc (mm_slot_cache);if (!slot)return ; +if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) { +mm_slot_free (mm_slot_cache, slot); +return ; + } +spin_lock (&khugepaged_mm_lock);mm_slot_insert (mm_slots_hash, mm, slot);/*
Quoted 5 lines
Arguably, on the race described above, likely the thread seeing the
MMF_VM_HUGEPAGEwould likely similarly have failed the allocation.I'm fine with either, just wanted to raise the (cleaner looking?) alternative where we just properly back off?
Yes this is also fine - I am overthinking but I wasn't going this way because ... A process doing THP allocations will fail on the mm_flags_test_and_set everytime after the first time. So we will have an unconditional overhead of mm_slot_alloc in the fault handler.
Again I am overthinking : )
Quoted 1 line
David Hildenbrand (Arm)
maintainerQuoted 40 lines · Dev Jain
Quoted 37 lines · David Hildenbrand (Arm)
Quoted 5 lines · Lance Yang
Right. A racing
khugepaged_enter_vma()can seeMMF_VM_HUGEPAGEis set and return, then !slot clears it again. If there is no laterkhugepaged_enter_vma(), the mm still wouldn't get registered :)So why not
mm/khugepaged.c +5
-2@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)/* __khugepaged_exit() must not run from under us */ VM_BUG_ON_MM (collapse_test_exit (mm), mm); -if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) -return ; slot =mm_slot_alloc (mm_slot_cache);if (!slot)return ; +if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) { +mm_slot_free (mm_slot_cache, slot); +return ; + } +spin_lock (&khugepaged_mm_lock);mm_slot_insert (mm_slots_hash, mm, slot);/* Arguably, on the race described above, likely the thread seeing the
MMF_VM_HUGEPAGEwould likely similarly have failed the allocation.
Quoted 6 lines
Quoted 2 lines
I'm fine with either, just wanted to raise the (cleaner looking?) alternative where we just properly back off?
Yes this is also fine - I am overthinking but I wasn't going this way because ... A process doing THP allocations will fail on the
mm_flags_test_and_seteverytime after the first time.
We should perform a mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) test before calling the function when the flag might not be set yet: in khugepaged_enter_vma()
khugepaged_fork() should only get called once per process.
Which makes sense, because mm_flags_test_and_set() is expensive.
Signature · 3 lines
Cheers, David
Dev Jain
reviewerQuoted 41 lines · David Hildenbrand (Arm)
Quoted 40 lines · Dev Jain
Quoted 37 lines · David Hildenbrand (Arm)
Quoted 5 lines · Lance Yang
Right. A racing
khugepaged_enter_vma()can seeMMF_VM_HUGEPAGEis set and return, then !slot clears it again. If there is no laterkhugepaged_enter_vma(), the mm still wouldn't get registered :)So why not
mm/khugepaged.c +5
-2@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)/* __khugepaged_exit() must not run from under us */ VM_BUG_ON_MM (collapse_test_exit (mm), mm); -if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) -return ; slot =mm_slot_alloc (mm_slot_cache);if (!slot)return ; +if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm))) { +mm_slot_free (mm_slot_cache, slot); +return ; + } +spin_lock (&khugepaged_mm_lock);mm_slot_insert (mm_slots_hash, mm, slot);/* Arguably, on the race described above, likely the thread seeing the
MMF_VM_HUGEPAGEwould likely similarly have failed the allocation.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I'm fine with either, just wanted to raise the (cleaner looking?) alternative where we just properly back off?
Yes this is also fine - I am overthinking but I wasn't going this way because ... A process doing THP allocations will fail on the
mm_flags_test_and_seteverytime after the first time.We should perform a
mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm)test before calling the function when the flag might not be set yet: inkhugepaged_enter_vma()
Ah that slipped my mind, you are right.
Quoted 5 lines
khugepaged_fork()should only get called once per process.Which makes sense, because
mm_flags_test_and_set()is expensive.