Skip to contentSkip to conversation

Possible race with page_maybe_dma_pinned?

DISCUSSION Peter Xu 4 participants 4 messages Sep 29, 2021 – Sep 30, 2021

Subsystems (2)
lore
Share & tools
lore

Peter Xu

author reviewer
#0 lore raw

Hi, all,

It seems to be racy to call page_maybe_dma_pinned() without properly taking the mm->write_protect_seq lock, which is taken read for fast gup.

Now we have 3 callers of page_maybe_dma_pinned():

1. page_needs_cow_for_dma
2. pte_is_pinned
3. shrink_page_list

The 1st one is good as it takes the seqlock for write properly. The 2nd & 3rd are missing, we may need to add them.

The race could trigger when the fast-gup of FOLL_PIN happened right after a call to page_maybe_dma_pinned() which returned false. One example for page reclaim of above case 3:

fast-gup thread page reclaim thread
--------------- -------------------
page_maybe_dma_pinned --> false put the page into swap cache
fast-gup with FOLL_PIN
unmap page in pgtables
...

So commit feb889fb40fa ("mm: don't put pinned pages into the swap cache", 2021-01-17) could still have a small window that will stop working.

Same thing to the pte_is_pinned for clear_refs, which is case 2nd above.

If anyone agrees, and if anyone would like to fix this, please add:

As this is originally spotted and reported by Andrea.

Thanks,

Signature · 1 line
Peter Xu
#1 lore raw
Quoted 9 lines · Peter Xu

Now we have 3 callers of page_maybe_dma_pinned():

1. page_needs_cow_for_dma
2. pte_is_pinned
3. shrink_page_list

The 1st one is good as it takes the seqlock for write properly. The 2nd & 3rd are missing, we may need to add them.

Well, the pte_is_pinned() case at least could do the seqlock in clear_soft_dirty() - it has the vma and mm available.

The page shrinker has always been problematic since it doesn't have the vm (and by "always" I mean "modern times" - long ago we used to scan virtually, in the days before rmap)

One option might be for fast-gup to give up on locked pages. I think the page lock is the only thing that shrink_page_list() serializes with.

Linus

reviewer
#2 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 7 lines · Peter Xu

Now we have 3 callers of page_maybe_dma_pinned():

1. page_needs_cow_for_dma
2. pte_is_pinned
3. shrink_page_list

Quoted 5 lines
Quoted 2 lines

The 1st one is good as it takes the seqlock for write properly. The 2nd & 3rd are missing, we may need to add them.

Well, the pte_is_pinned() case at least could do the seqlock in clear_soft_dirty() - it has the vma and mm available.

That part seems straightforward, OK.

Quoted 9 lines

The page shrinker has always been problematic since it doesn't have the vm (and by "always" I mean "modern times" - long ago we used to scan virtually, in the days before rmap)

One option might be for fast-gup to give up on locked pages. I think the page lock is the only thing that shrink_page_list() serializes with.

In order to avoid locked pages in gup fast, it is easiest to do a check for locked pages *after* fast-pinning them, and unpin them before returning to the caller. This makes the change much smaller.

However, doing so leaves a window of time during which the pages are still marked as maybe-dma-pinned, although those pages are never returned to the caller as such. There is already code that is subject to this in lockless_pages_from_mm(), for the case of a failed seqlock read. I'm thinking it's probably OK, because the pages are not going to be held long-term. They will be unpinned before returning from lockless_pages_from_mm().

The counter argument is that this is merely making the race window smaller, which is usually something that I argue against because it just leads to harder-to-find bugs...

To be specific, here's what I mean:

mm/gup.c +17 -0
index 886d6148d3d03..8ba871a927668 100644
@@ -2657,6 +2657,7 @@ static unsigned long lockless_pages_from_mm(unsigned long start,
 {
 	unsigned long flags;
 	int nr_pinned = 0;
+	int i;
 	unsigned seq;
 
 	if (!IS_ENABLED(CONFIG_HAVE_FAST_GUP) ||
@@ -2693,7 +2694,23 @@ static unsigned long lockless_pages_from_mm(unsigned long start,
 			unpin_user_pages(pages, nr_pinned);
 			return 0;
 		}
+
+		/*
+		 * Avoiding locked pages, in this fast/lockless context, will
+		 * avoid interfering with shrink_page_list(), in particular.
+		 * Give up upon finding the first locked page, but keep the
+		 * earlier pages, so that slow gup does not have to re-pin them.
+		 */
+		for (i = 0; i < nr_pinned; i++)  {
+			if (PageLocked(pages[i])) {
+				unpin_user_pages(&pages[i], nr_pinned - i);
+				nr_pinned = i + 1;
+				break;
+			}
+		}
 	}
+
+
 	return nr_pinned;
 }
 

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#3 lore raw
Quoted 39 lines · John Hubbard
Quoted 8 lines · Linus Torvalds
Quoted 7 lines · Peter Xu

Now we have 3 callers of page_maybe_dma_pinned():

1. page_needs_cow_for_dma
2. pte_is_pinned
3. shrink_page_list

Quoted 5 lines
Quoted 2 lines

The 1st one is good as it takes the seqlock for write properly. The 2nd & 3rd are missing, we may need to add them.

Well, the pte_is_pinned() case at least could do the seqlock in clear_soft_dirty() - it has the vma and mm available.

That part seems straightforward, OK.

Quoted 9 lines

The page shrinker has always been problematic since it doesn't have the vm (and by "always" I mean "modern times" - long ago we used to scan virtually, in the days before rmap)

One option might be for fast-gup to give up on locked pages. I think the page lock is the only thing that shrink_page_list() serializes with.

In order to avoid locked pages in gup fast, it is easiest to do a check for locked pages *after* fast-pinning them, and unpin them before returning to the caller. This makes the change much smaller.

However, doing so leaves a window of time during which the pages are still marked as maybe-dma-pinned, although those pages are never returned to the caller as such. There is already code that is subject to this in lockless_pages_from_mm(), for the case of a failed seqlock read. I'm thinking it's probably OK, because the pages are not going to be held long-term. They will be unpinned before returning from lockless_pages_from_mm().

Quoted 3 lines

The counter argument is that this is merely making the race window smaller, which is usually something that I argue against because it just leads to harder-to-find bugs...

Yeah, what you propose actually does not guarantee that the reclaim and page pinning cannot race in a way that the page gets unmapped and gup_fast() returns a pinned page. Which is what the code in shrink_page_list() tries to prevent AFAIU.

The only place where I can see us doing a sanely race-free check in shrink_page_list() path is inside try_to_unmap() - we could implement unmap_if_not_pinned semantics and inside the rmap walk we can bump up the seqcounts (it even conceptually makes some sense) to make sure page_maybe_dma_pinned() check we'd do during the rmap walk is not racing with pup_fast(). By the time we leave the seqcount critical section, the page will be unmapped so we can be sure there will be no new pins of the page.

Honza

Quoted 44 lines

To be specific, here's what I mean:

mm/gup.c +17 -0
index 886d6148d3d03..8ba871a927668 100644
@@ -2657,6 +2657,7 @@ static unsigned long lockless_pages_from_mm(unsigned long start,
 {
 	unsigned long flags;
 	int nr_pinned = 0;
+	int i;
 	unsigned seq;
 
 	if (!IS_ENABLED(CONFIG_HAVE_FAST_GUP) ||
@@ -2693,7 +2694,23 @@ static unsigned long lockless_pages_from_mm(unsigned long start,
 			unpin_user_pages(pages, nr_pinned);
 			return 0;
 		}
+
+		/*
+		 * Avoiding locked pages, in this fast/lockless context, will
+		 * avoid interfering with shrink_page_list(), in particular.
+		 * Give up upon finding the first locked page, but keep the
+		 * earlier pages, so that slow gup does not have to re-pin them.
+		 */
+		for (i = 0; i < nr_pinned; i++)  {
+			if (PageLocked(pages[i])) {
+				unpin_user_pages(&pages[i], nr_pinned - i);
+				nr_pinned = i + 1;
+				break;
+			}
+		}
 	}
+
+
 	return nr_pinned;
 }
 

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR