Skip to contentSkip to conversation

userfaultfd: do not block on locking a large folio with raised refcount

PATCH In review Suren Baghdasaryan 5 participants 12 messages Feb 25, 2025 – Feb 26, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 userfaultfd: do not block on locking a large folio with raised refcount

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +16 -1
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
 		 */
 		if (!src_folio) {
 			struct folio *folio;
+			bool locked;
 
 			/*
 			 * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
 				goto out;
 			}
 
+			locked = folio_trylock(folio);
+			/*
+			 * We avoid waiting for folio lock with a raised refcount
+			 * for large folios because extra refcounts will result in
+			 * split_folio() failing later and retrying. If multiple
+			 * tasks are trying to move a large folio we can end
+			 * livelocking.
+			 */
+			if (!locked && folio_test_large(folio)) {
+				spin_unlock(src_ptl);
+				err = -EAGAIN;
+				goto out;
+			}
+
 			folio_get(folio);
 			src_folio = folio;
 			src_folio_pte = orig_src_pte;
 			spin_unlock(src_ptl);
 
-			if (!folio_trylock(src_folio)) {
+			if (!locked) {
 				pte_unmap(&orig_src_pte);
 				pte_unmap(&orig_dst_pte);
 				src_pte = dst_pte = NULL;

base-commit: 801d47bd96ce22acd43809bc09e004679f707c39

Signature · 1 line
2.48.1.658.g4767266eb4-goog
#1 lore raw
Quoted 25 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

Quoted 2 lines

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

Thanks so much for fixing this, Suren.

Fixes adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Quoted 51 lines
---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +16 -1
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
                 */
                if (!src_folio) {
                        struct folio *folio;
+                       bool locked;
 
                        /*
                         * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
                                goto out;
                        }
 
+                       locked = folio_trylock(folio);
+                       /*
+                        * We avoid waiting for folio lock with a raised refcount
+                        * for large folios because extra refcounts will result in
+                        * split_folio() failing later and retrying. If multiple
+                        * tasks are trying to move a large folio we can end
+                        * livelocking.
+                        */
+                       if (!locked && folio_test_large(folio)) {
+                               spin_unlock(src_ptl);
+                               err = -EAGAIN;
+                               goto out;
+                       }
+
                        folio_get(folio);
                        src_folio = folio;
                        src_folio_pte = orig_src_pte;
                        spin_unlock(src_ptl);
 
-                       if (!folio_trylock(src_folio)) {
+                       if (!locked) {
                                pte_unmap(&orig_src_pte);
                                pte_unmap(&orig_dst_pte);
                                src_pte = dst_pte = NULL;

base-commit: 801d47bd96ce22acd43809bc09e004679f707c39 --
2.48.1.658.g4767266eb4-goog

reviewer Reviewed-by on v1 1/1
#2 lore raw
In reply to Suren Baghdasaryan
Quoted 24 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

Quoted 4 lines

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

One question irrelevant of this change below..

Quoted 33 lines
---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +14 -0
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
 		 */
 		if (!src_folio) {
 			struct folio *folio;
+			bool locked;
 
 			/*
 			 * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
 				goto out;
 			}
 
+			locked = folio_trylock(folio);
+			/*
+			 * We avoid waiting for folio lock with a raised refcount
+			 * for large folios because extra refcounts will result in
+			 * split_folio() failing later and retrying. If multiple
+			 * tasks are trying to move a large folio we can end
+			 * livelocking.
+			 */
+			if (!locked && folio_test_large(folio)) {
+				spin_unlock(src_ptl);
+				err = -EAGAIN;
+				goto out;
+			}

mm/userfaultfd.c static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, · line 1280

Quoted 10 lines
mm/userfaultfd.c +2 -1
+
 			folio_get(folio);
 			src_folio = folio;
 			src_folio_pte = orig_src_pte;
 			spin_unlock(src_ptl);
 
-			if (!folio_trylock(src_folio)) {
+			if (!locked) {
 				pte_unmap(&orig_src_pte);
 				pte_unmap(&orig_dst_pte);

.. just notice this. Are these problematic? I mean, orig_*_pte are stack variables, afaict. I'd expect these things blow on HIGHPTE..

Quoted 6 lines
mm/userfaultfd.c +0 -0
 				src_pte = dst_pte = NULL;

base-commit: 801d47bd96ce22acd43809bc09e004679f707c39

Signature · 1 line
2.48.1.658.g4767266eb4-goog
Signature · 1 line
Peter Xu
#3 lore raw
Quoted 74 lines · Peter Xu
Quoted 24 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

Quoted 4 lines

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

One question irrelevant of this change below..

Quoted 39 lines
---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +15 -0
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
               */
              if (!src_folio) {
                      struct folio *folio;
+                     bool locked;
 
                      /*
                       * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
                              goto out;
                      }
 
+                     locked = folio_trylock(folio);
+                     /*
+                      * We avoid waiting for folio lock with a raised refcount
+                      * for large folios because extra refcounts will result in
+                      * split_folio() failing later and retrying. If multiple
+                      * tasks are trying to move a large folio we can end
+                      * livelocking.
+                      */
+                     if (!locked && folio_test_large(folio)) {
+                             spin_unlock(src_ptl);
+                             err = -EAGAIN;
+                             goto out;
+                     }
+
                      folio_get(folio);
                      src_folio = folio;
                      src_folio_pte = orig_src_pte;
                      spin_unlock(src_ptl);
 
Quoted 7 lines

mm/userfaultfd.c static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, · line 1280

Quoted 4 lines
mm/userfaultfd.c +1 -1
-                     if (!folio_trylock(src_folio)) {
+                     if (!locked) {
                              pte_unmap(&orig_src_pte);
                              pte_unmap(&orig_dst_pte);

.. just notice this. Are these problematic? I mean, orig_*_pte are stack variables, afaict. I'd expect these things blow on HIGHPTE..

Ugh! Yes, I think so. From a quick look, move_pages_pte() is the only place we have this issue and I don't see a reason for copying src_pte and dst_pte values. I'll spend some more time trying to understand if we really need these local copies.

Quoted 11 lines
Quoted 7 lines
mm/userfaultfd.c +0 -0
                              src_pte = dst_pte = NULL;

base-commit: 801d47bd96ce22acd43809bc09e004679f707c39 --
2.48.1.658.g4767266eb4-goog

--
Peter Xu

#4 lore raw
Quoted 84 lines · Suren Baghdasaryan
Quoted 74 lines · Peter Xu
Quoted 24 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

Quoted 4 lines

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

One question irrelevant of this change below..

Quoted 39 lines
---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +15 -0
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
               */
              if (!src_folio) {
                      struct folio *folio;
+                     bool locked;
 
                      /*
                       * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
                              goto out;
                      }
 
+                     locked = folio_trylock(folio);
+                     /*
+                      * We avoid waiting for folio lock with a raised refcount
+                      * for large folios because extra refcounts will result in
+                      * split_folio() failing later and retrying. If multiple
+                      * tasks are trying to move a large folio we can end
+                      * livelocking.
+                      */
+                     if (!locked && folio_test_large(folio)) {
+                             spin_unlock(src_ptl);
+                             err = -EAGAIN;
+                             goto out;
+                     }
+
                      folio_get(folio);
                      src_folio = folio;
                      src_folio_pte = orig_src_pte;
                      spin_unlock(src_ptl);
 
Quoted 7 lines

mm/userfaultfd.c static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, · line 1280

Quoted 4 lines
mm/userfaultfd.c +1 -1
-                     if (!folio_trylock(src_folio)) {
+                     if (!locked) {
                              pte_unmap(&orig_src_pte);
                              pte_unmap(&orig_dst_pte);

.. just notice this. Are these problematic? I mean, orig_*_pte are stack variables, afaict. I'd expect these things blow on HIGHPTE..

Quoted 4 lines

Ugh! Yes, I think so. From a quick look, move_pages_pte() is the only place we have this issue and I don't see a reason for copying src_pte and dst_pte values. I'll spend some more time trying to understand if we really need these local copies.

Ah, we copy the values to later check if PTEs changed from under us. But I see no reason we need to use orig_{src|dst}_pte instead of {src|dst}_pte when doing pte_unmap(). I think we can safely replace them with the original ones. WDYT?

Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
mm/userfaultfd.c +0 -0
                              src_pte = dst_pte = NULL;

base-commit: 801d47bd96ce22acd43809bc09e004679f707c39 --
2.48.1.658.g4767266eb4-goog

--
Peter Xu

reviewer
#5 lore raw
Quoted 85 lines · Suren Baghdasaryan
Quoted 84 lines · Suren Baghdasaryan
Quoted 81 lines · Peter Xu
Quoted 24 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

Quoted 4 lines

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

One question irrelevant of this change below..

Quoted 33 lines
---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +14 -0
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
               */
              if (!src_folio) {
                      struct folio *folio;
+                     bool locked;
 
                      /*
                       * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
                              goto out;
                      }
 
+                     locked = folio_trylock(folio);
+                     /*
+                      * We avoid waiting for folio lock with a raised refcount
+                      * for large folios because extra refcounts will result in
+                      * split_folio() failing later and retrying. If multiple
+                      * tasks are trying to move a large folio we can end
+                      * livelocking.
+                      */
+                     if (!locked && folio_test_large(folio)) {
+                             spin_unlock(src_ptl);
+                             err = -EAGAIN;
+                             goto out;
+                     }

mm/userfaultfd.c static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, · line 1280

Quoted 10 lines
mm/userfaultfd.c +2 -1
+
                      folio_get(folio);
                      src_folio = folio;
                      src_folio_pte = orig_src_pte;
                      spin_unlock(src_ptl);
 
-                     if (!folio_trylock(src_folio)) {
+                     if (!locked) {
                              pte_unmap(&orig_src_pte);
                              pte_unmap(&orig_dst_pte);

.. just notice this. Are these problematic? I mean, orig_*_pte are stack variables, afaict. I'd expect these things blow on HIGHPTE..

Quoted 8 lines
Quoted 4 lines

Ugh! Yes, I think so. From a quick look, move_pages_pte() is the only place we have this issue and I don't see a reason for copying src_pte and dst_pte values. I'll spend some more time trying to understand if we really need these local copies.

Ah, we copy the values to later check if PTEs changed from under us. But I see no reason we need to use orig_{src|dst}_pte instead of {src|dst}_pte when doing pte_unmap(). I think we can safely replace

That looks like something we just overlooked before, meanwhile it's undetectable on !HIGHPTE anyway.. in which case the addr ignored, and that turns always into an rcu unlock.

Quoted 1 line

them with the original ones. WDYT?

Agreed. Maybe not "the original ones" if we're looking for words to put into the commit message: it could be "we should kunmap() whatever we kmap()ed before", or something better.

Thanks,

Signature · 1 line
Peter Xu
#6 lore raw
Quoted 100 lines · Peter Xu
Quoted 85 lines · Suren Baghdasaryan
Quoted 84 lines · Suren Baghdasaryan
Quoted 81 lines · Peter Xu
Quoted 24 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

Quoted 4 lines

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

One question irrelevant of this change below..

Quoted 33 lines
---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +14 -0
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
               */
              if (!src_folio) {
                      struct folio *folio;
+                     bool locked;
 
                      /*
                       * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
                              goto out;
                      }
 
+                     locked = folio_trylock(folio);
+                     /*
+                      * We avoid waiting for folio lock with a raised refcount
+                      * for large folios because extra refcounts will result in
+                      * split_folio() failing later and retrying. If multiple
+                      * tasks are trying to move a large folio we can end
+                      * livelocking.
+                      */
+                     if (!locked && folio_test_large(folio)) {
+                             spin_unlock(src_ptl);
+                             err = -EAGAIN;
+                             goto out;
+                     }

mm/userfaultfd.c static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, · line 1280

Quoted 10 lines
mm/userfaultfd.c +2 -1
+
                      folio_get(folio);
                      src_folio = folio;
                      src_folio_pte = orig_src_pte;
                      spin_unlock(src_ptl);
 
-                     if (!folio_trylock(src_folio)) {
+                     if (!locked) {
                              pte_unmap(&orig_src_pte);
                              pte_unmap(&orig_dst_pte);

.. just notice this. Are these problematic? I mean, orig_*_pte are stack variables, afaict. I'd expect these things blow on HIGHPTE..

Quoted 8 lines
Quoted 4 lines

Ugh! Yes, I think so. From a quick look, move_pages_pte() is the only place we have this issue and I don't see a reason for copying src_pte and dst_pte values. I'll spend some more time trying to understand if we really need these local copies.

Ah, we copy the values to later check if PTEs changed from under us. But I see no reason we need to use orig_{src|dst}_pte instead of {src|dst}_pte when doing pte_unmap(). I think we can safely replace

That looks like something we just overlooked before, meanwhile it's undetectable on !HIGHPTE anyway.. in which case the addr ignored, and that turns always into an rcu unlock.

Quoted 5 lines
Quoted 1 line

them with the original ones. WDYT?

Agreed. Maybe not "the original ones" if we're looking for words to put into the commit message: it could be "we should kunmap() whatever we kmap()ed before", or something better.

Sounds good to me. I'll give others one day to provide their input and will post a fix.
Thanks!

Quoted 6 lines

Thanks,

--
Peter Xu

#7 lore raw
In reply to Suren Baghdasaryan
Quoted 52 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +5 -0
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
 		 */
 		if (!src_folio) {
 			struct folio *folio;
+			bool locked;
 
 			/*
 			 * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
 				goto out;
 			}
 
+			locked = folio_trylock(folio);
+			/*
+			 * We avoid waiting for folio lock with a raised refcount
+			 * for large folios because extra refcounts will result in

mm/userfaultfd.c static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, · line 1272

Quoted 10 lines
mm/userfaultfd.c +10 -0
+			 * split_folio() failing later and retrying. If multiple
+			 * tasks are trying to move a large folio we can end
+			 * livelocking.
+			 */
+			if (!locked && folio_test_large(folio)) {
+				spin_unlock(src_ptl);
+				err = -EAGAIN;
+				goto out;
+			}
+

Reversing the locking/folio_get() is okay because of the src_ptl spin lock, right? It might be worth saying something about it in the comment?

Quoted 15 lines
mm/userfaultfd.c +1 -1
 			folio_get(folio);
 			src_folio = folio;
 			src_folio_pte = orig_src_pte;
 			spin_unlock(src_ptl);
 
-			if (!folio_trylock(src_folio)) {
+			if (!locked) {
 				pte_unmap(&orig_src_pte);
 				pte_unmap(&orig_dst_pte);
 				src_pte = dst_pte = NULL;

base-commit: 801d47bd96ce22acd43809bc09e004679f707c39

Signature · 1 line
2.48.1.658.g4767266eb4-goog
#8 lore raw
Quoted 60 lines · Liam R. Howlett
Quoted 58 lines · Suren Baghdasaryan

Lokesh recently raised an issue about UFFDIO_MOVE getting into a deadlock state when it goes into split_folio() with raised folio refcount. split_folio() expects the reference count to be exactly mapcount + num_pages_in_folio + 1 (see can_split_folio()) and fails with EAGAIN otherwise. If multiple processes are trying to move the same large folio, they raise the refcount (all tasks succeed in that) then one of them succeeds in locking the folio, while others will block in folio_lock() while keeping the refcount raised. The winner of this race will proceed with calling split_folio() and will fail returning EAGAIN to the caller and unlocking the folio. The next competing process will get the folio locked and will go through the same flow. In the meantime the original winner will be retried and will block in folio_lock(), getting into the queue of waiting processes only to repeat the same path. All this results in a livelock.
An easy fix would be to avoid waiting for the folio lock while holding folio refcount, similar to madvise_free_huge_pmd() where folio lock is acquired before raising the folio refcount.
Modify move_pages_pte() to try locking the folio first and if that fails and the folio is large then return EAGAIN without touching the folio refcount. If the folio is single-page then split_folio() is not called, so we don't have this issue.
Lokesh has a reproducer [1] and I verified that this change fixes the issue.

[1] https://github.com/lokeshgidra/uffd_move_ioctl_deadlock

---
 mm/userfaultfd.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
mm/userfaultfd.c +11 -0
index 867898c4e30b..f17f8290c523 100644
@@ -1236,6 +1236,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
               */
              if (!src_folio) {
                      struct folio *folio;
+                     bool locked;
 
                      /*
                       * Pin the page while holding the lock to be sure the
@@ -1255,12 +1256,26 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,
                              goto out;
                      }
 
+                     locked = folio_trylock(folio);
+                     /*
+                      * We avoid waiting for folio lock with a raised refcount
+                      * for large folios because extra refcounts will result in
+                      * split_folio() failing later and retrying. If multiple
+                      * tasks are trying to move a large folio we can end
+                      * livelocking.
+                      */
+                     if (!locked && folio_test_large(folio)) {
+                             spin_unlock(src_ptl);
Quoted 8 lines

mm/userfaultfd.c static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, · line 1272

Quoted 4 lines
mm/userfaultfd.c +4 -0
+                             err = -EAGAIN;
+                             goto out;
+                     }
+

Reversing the locking/folio_get() is okay because of the src_ptl spin lock, right? It might be worth saying something about it in the comment?

That is correct. We take both folio lock and refcount before we drop PTL. I'll add a comment. Thanks!

Quoted 16 lines
Quoted 16 lines
mm/userfaultfd.c +1 -1
                      folio_get(folio);
                      src_folio = folio;
                      src_folio_pte = orig_src_pte;
                      spin_unlock(src_ptl);
 
-                     if (!folio_trylock(src_folio)) {
+                     if (!locked) {
                              pte_unmap(&orig_src_pte);
                              pte_unmap(&orig_dst_pte);
                              src_pte = dst_pte = NULL;

base-commit: 801d47bd96ce22acd43809bc09e004679f707c39 --
2.48.1.658.g4767266eb4-goog

#9 lore raw
Quoted 7 lines · Suren Baghdasaryan
Quoted 3 lines · Liam R. Howlett

Reversing the locking/folio_get() is okay because of the src_ptl spin lock, right? It might be worth saying something about it in the comment?

That is correct. We take both folio lock and refcount before we drop PTL. I'll add a comment. Thanks!

In the commit message, not in the code, please.

#10 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 7 lines · Suren Baghdasaryan
Quoted 3 lines · Liam R. Howlett

Reversing the locking/folio_get() is okay because of the src_ptl spin lock, right? It might be worth saying something about it in the comment?

That is correct. We take both folio lock and refcount before we drop PTL. I'll add a comment. Thanks!

In the commit message, not in the code, please.

Ack.

#11 lore raw
Quoted 9 lines · Suren Baghdasaryan
Quoted 7 lines · Matthew Wilcox
Quoted 5 lines · Suren Baghdasaryan
Quoted 3 lines · Liam R. Howlett

Reversing the locking/folio_get() is okay because of the src_ptl spin lock, right? It might be worth saying something about it in the comment?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

That is correct. We take both folio lock and refcount before we drop PTL. I'll add a comment. Thanks!

In the commit message, not in the code, please.

Ack.

I posted v2 at https://lore.kernel.org/all/[email protected]/
as part of a pachset which includes the fix for PTE unmapping that
Peter reported. Patchset is rebased over mm-hotfixes-unstable which includes Barry's fix to the nearby code. This avoids merge conflicts.