Skip to contentSkip to conversation

vhost, docs: convert to pin_user_pages(), new "case 5"

PATCH Superseded John Hubbard 5 participants 9 messages May 29, 2020 – Jun 12, 2020

Subsystems (8)

v1 of 2 Superseded 1 of 2 applied Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover vhost, docs: convert to pin_user_pages(), new "case 5"

Hi,

It recently became clear to me that there are some get_user_pages*() callers that don't fit neatly into any of the four cases that are so far listed in pin_user_pages.rst. vhost.c is one of those.

Add a Case 5 to the documentation, and refer to that when converting vhost.c.

Thanks to Jan Kara for helping me (again) in understanding the interaction between get_user_pages() and page writeback [1].

This is based on today's mmotm, which has a nearby patch to pin_user_pages.rst that rewords cases 3 and 4.

Note that I have only compile-tested the vhost.c patch, although that does also include cross-compiling for a few other arches. Any run-time testing would be greatly appreciated.

[1] https://lore.kernel.org/r/[email protected]

John Hubbard (2):
docs: mm/gup: pin_user_pages.rst: add a "case 5"
vhost: convert get_user_pages() --> pin_user_pages()

 Documentation/core-api/pin_user_pages.rst | 20 ++++++++++++++++++++
 drivers/vhost/vhost.c                     |  5 ++---
 2 files changed, 22 insertions(+), 3 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/2 docs: mm/gup: pin_user_pages.rst: add a "case 5"

There are four cases listed in pin_user_pages.rst. These are intended to help developers figure out whether to use get_user_pages*(), or pin_user_pages*(). However, the four cases do not cover all the situations. For example, drivers/vhost/vhost.c has a "pin, write to page, set page dirty, unpin" case.

Add a fifth case, to help explain that there is a general pattern that requires pin_user_pages*() API calls.

Cc Vlastimil Babka <[email protected]>
Cc Jan Kara <[email protected]>
Cc Jérôme Glisse <[email protected]>
Cc Dave Chinner <[email protected]>
Cc Jonathan Corbet <[email protected]>
---
 Documentation/core-api/pin_user_pages.rst | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
Documentation/core-api/pin_user_pages.rst +20 -0
index 4675b04e8829..b9f2688a2c67 100644
@@ -171,6 +171,26 @@ If only struct page data (as opposed to the actual memory contents that a page
 is tracking) is affected, then normal GUP calls are sufficient, and neither flag
 needs to be set.
 
+CASE 5: Pinning in order to write to the data within the page
+-------------------------------------------------------------
+Even though neither DMA nor Direct IO is involved, just a simple case of "pin,
+access page's data, unpin" can cause a problem. Case 5 may be considered a
+superset of Case 1, plus Case 2, plus anything that invokes that pattern. In
+other words, if the code is neither Case 1 nor Case 2, it may still require
+FOLL_PIN, for patterns like this:
+
+Correct (uses FOLL_PIN calls):
+    pin_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    unpin_user_pages()
+
+INCORRECT (uses FOLL_GET calls):
+    get_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    put_page()
+
 page_maybe_dma_pinned(): the whole point of pinning
 ===================================================
 
Signature · 6 lines
2.26.2

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#2 lore raw

Re: [PATCH 1/2] docs: mm/gup: pin_user_pages.rst: add a "case 5"

Quoted 27 lines · John Hubbard

There are four cases listed in pin_user_pages.rst. These are intended to help developers figure out whether to use get_user_pages*(), or pin_user_pages*(). However, the four cases do not cover all the situations. For example, drivers/vhost/vhost.c has a "pin, write to page, set page dirty, unpin" case.

Add a fifth case, to help explain that there is a general pattern that requires pin_user_pages*() API calls.

Cc Vlastimil Babka <[email protected]>
Cc Jan Kara <[email protected]>
Cc Jérôme Glisse <[email protected]>
Cc Dave Chinner <[email protected]>
Cc Jonathan Corbet <[email protected]>
---
 Documentation/core-api/pin_user_pages.rst | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
Documentation/core-api/pin_user_pages.rst +0 -0
index 4675b04e8829..b9f2688a2c67 100644

Documentation/core-api/pin_user_pages.rst If only struct page data (as opposed to the actual memory contents that a page · line 177

Quoted 7 lines
Documentation/core-api/pin_user_pages.rst +4 -0
@@ -171,6 +171,26 @@ If only struct page data (as opposed to the actual memory contents that a page
 is tracking) is affected, then normal GUP calls are sufficient, and neither flag
 needs to be set.
 
+CASE 5: Pinning in order to write to the data within the page
+-------------------------------------------------------------
+Even though neither DMA nor Direct IO is involved, just a simple case of "pin,
+access page's data, unpin" can cause a problem.

Will it be, *"pin, access page's data, set page dirty, unpin" * ?

Case 5 may be considered a

Quoted 22 lines
Documentation/core-api/pin_user_pages.rst +16 -1
+superset of Case 1, plus Case 2, plus anything that invokes that pattern. In
+other words, if the code is neither Case 1 nor Case 2, it may still require
+FOLL_PIN, for patterns like this:
+
+Correct (uses FOLL_PIN calls):
+    pin_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    unpin_user_pages()
+
+INCORRECT (uses FOLL_GET calls):
+    get_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    put_page()
+
 page_maybe_dma_pinned(): the whole point of pinning
 ===================================================
 
--

2.26.2

#3 lore raw

Re: [PATCH 1/2] docs: mm/gup: pin_user_pages.rst: add a "case 5"

On 2020-05-31 00:11, Souptick Joarder wrote:
...

Quoted 8 lines
Quoted 8 lines
Documentation/core-api/pin_user_pages.rst +0 -0
index 4675b04e8829..b9f2688a2c67 100644
@@ -171,6 +171,26 @@ If only struct page data (as opposed to the actual memory contents that a page
  is tracking) is affected, then normal GUP calls are sufficient, and neither flag
  needs to be set.
 
Quoted 6 lines

Documentation/core-api/pin_user_pages.rst If only struct page data (as opposed to the actual memory contents that a page

Quoted 4 lines
Documentation/core-api/pin_user_pages.rst +4 -0
+CASE 5: Pinning in order to write to the data within the page
+-------------------------------------------------------------
+Even though neither DMA nor Direct IO is involved, just a simple case of "pin,
+access page's data, unpin" can cause a problem.

Will it be, *"pin, access page's data, set page dirty, unpin" * ?

Well...the problem can show up with just accessing (writing) the data. But it is true that this statement is a little different from the patterns below, which is confusing. I'll delete set_page_dirty() from each of them, in order to avoid confusing things. (Although each is correct.) And I'll also change the above to "pin, write to a page's data, upin".

set_page_dirty() interactions are really just extra credit here. :) And fully read-only situations won't cause a problem.

Quoted 18 lines

Case 5 may be considered a

Quoted 6 lines
Documentation/core-api/pin_user_pages.rst +6 -0
+superset of Case 1, plus Case 2, plus anything that invokes that pattern. In
+other words, if the code is neither Case 1 nor Case 2, it may still require
+FOLL_PIN, for patterns like this:
+
+Correct (uses FOLL_PIN calls):
+    pin_user_pages()

Documentation/core-api/pin_user_pages.rst If only struct page data (as opposed to the actual memory contents that a page · line 193

Quoted 10 lines
Documentation/core-api/pin_user_pages.rst +10 -0
+    access the data within the pages
+    set_page_dirty_lock()
+    unpin_user_pages()
+
+INCORRECT (uses FOLL_GET calls):
+    get_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    put_page()
+

I'll send a v2 shortly.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#4 lore raw

Re: [PATCH 1/2] docs: mm/gup: pin_user_pages.rst: add a "case 5"

In reply to John Hubbard
Quoted 9 lines · John Hubbard
Documentation/core-api/pin_user_pages.rst +9 -0
+CASE 5: Pinning in order to write to the data within the page
+-------------------------------------------------------------
+Even though neither DMA nor Direct IO is involved, just a simple case of "pin,
+access page's data, unpin" can cause a problem. Case 5 may be considered a
+superset of Case 1, plus Case 2, plus anything that invokes that pattern. In
+other words, if the code is neither Case 1 nor Case 2, it may still require
+FOLL_PIN, for patterns like this:
+
+Correct (uses FOLL_PIN calls):

Documentation/core-api/pin_user_pages.rst If only struct page data (as opposed to the actual memory contents that a page · line 192

Quoted 10 lines
Documentation/core-api/pin_user_pages.rst +10 -0
+    pin_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    unpin_user_pages()
+
+INCORRECT (uses FOLL_GET calls):
+    get_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    put_page()

Why does this case need to pin? Why can't it just do ...

	get_user_pages()
	lock_page(page);
	... modify the data ...
	set_page_dirty(page);
	unlock_page(page);
#5 lore raw

Re: [PATCH 1/2] docs: mm/gup: pin_user_pages.rst: add a "case 5"

Quoted 23 lines · Matthew Wilcox
Quoted 9 lines · John Hubbard
Documentation/core-api/pin_user_pages.rst +9 -0
+CASE 5: Pinning in order to write to the data within the page
+-------------------------------------------------------------
+Even though neither DMA nor Direct IO is involved, just a simple case of "pin,
+access page's data, unpin" can cause a problem. Case 5 may be considered a
+superset of Case 1, plus Case 2, plus anything that invokes that pattern. In
+other words, if the code is neither Case 1 nor Case 2, it may still require
+FOLL_PIN, for patterns like this:
+
+Correct (uses FOLL_PIN calls):

Documentation/core-api/pin_user_pages.rst If only struct page data (as opposed to the actual memory contents that a page · line 192

Quoted 10 lines
Documentation/core-api/pin_user_pages.rst +10 -0
+    pin_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    unpin_user_pages()
+
+INCORRECT (uses FOLL_GET calls):
+    get_user_pages()
+    access the data within the pages
+    set_page_dirty_lock()
+    put_page()

Why does this case need to pin? Why can't it just do ...

Quoted 6 lines
	get_user_pages()
	lock_page(page);
	... modify the data ...
	set_page_dirty(page);
	unlock_page(page);

Yes, it could do that. And that would also make a good additional "correct" example. Especially for the case of just dealing with a single page, lock_page() has the benefit of completely fixing the problem *today*, without waiting for the pin_user_pages*() handling improvements to get implemented.

And it's also another (probably better) way to fix the vhost.c problem, than commit 690623e1b496 ("vhost: convert get_user_pages() --> pin_user_pages()").

I'm inclined to leave vhost.c alone for now, unless someone really prefers it to be changed, but to update the Case 5 documentation with your point above. Sound about right?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#6 lore raw

PATCH 2/2 vhost: convert get_user_pages() --> pin_user_pages()

In reply to John Hubbard

This code was using get_user_pages*(), in approximately a "Case 5" scenario (accessing the data within a page), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.

There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.

[1] Documentation/core-api/pin_user_pages.rst

[2] "Explicit pinning of user-space pages":

    https://lwn.net/Articles/807108/
---
 drivers/vhost/vhost.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/vhost/vhost.c +2 -3
index 21a59b598ed8..596132a96cd5 100644
@@ -1762,15 +1762,14 @@ static int set_bit_to_user(int nr, void __user *addr)
 	int bit = nr + (log % PAGE_SIZE) * 8;
 	int r;
 
-	r = get_user_pages_fast(log, 1, FOLL_WRITE, &page);
+	r = pin_user_pages_fast(log, 1, FOLL_WRITE, &page);
 	if (r < 0)
 		return r;
 	BUG_ON(r != 1);
 	base = kmap_atomic(page);
 	set_bit(bit, base);
 	kunmap_atomic(base);
-	set_page_dirty_lock(page);
-	put_page(page);
+	unpin_user_pages_dirty_lock(&page, 1, true);
 	return 0;
 }
 
Signature · 1 line
2.26.2
reviewer Reviewed-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] vhost: convert get_user_pages() --> pin_user_pages()

Quoted 12 lines · John Hubbard

This code was using get_user_pages*(), in approximately a "Case 5" scenario (accessing the data within a page), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.

There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.

[1] Documentation/core-api/pin_user_pages.rst

[2] "Explicit pinning of user-space pages":

Quoted 8 lines
    https://lwn.net/Articles/807108/

Looks good to me. You can add:

Honza

Quoted 29 lines
---
 drivers/vhost/vhost.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/vhost/vhost.c +2 -3
index 21a59b598ed8..596132a96cd5 100644
@@ -1762,15 +1762,14 @@ static int set_bit_to_user(int nr, void __user *addr)
 	int bit = nr + (log % PAGE_SIZE) * 8;
 	int r;
 
-	r = get_user_pages_fast(log, 1, FOLL_WRITE, &page);
+	r = pin_user_pages_fast(log, 1, FOLL_WRITE, &page);
 	if (r < 0)
 		return r;
 	BUG_ON(r != 1);
 	base = kmap_atomic(page);
 	set_bit(bit, base);
 	kunmap_atomic(base);
-	set_page_dirty_lock(page);
-	put_page(page);
+	unpin_user_pages_dirty_lock(&page, 1, true);
 	return 0;
 }
 
Signature · 1 line
2.26.2
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer Acked-by on v1 2/2
#8 lore raw

Re: [PATCH 2/2] vhost: convert get_user_pages() --> pin_user_pages()

In reply to John Hubbard
Quoted 12 lines · John Hubbard

This code was using get_user_pages*(), in approximately a "Case 5" scenario (accessing the data within a page), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.

There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.

[1] Documentation/core-api/pin_user_pages.rst

[2] "Explicit pinning of user-space pages":

Quoted 8 lines
    https://lwn.net/Articles/807108/
Quoted 28 lines
---
 drivers/vhost/vhost.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/vhost/vhost.c +2 -3
index 21a59b598ed8..596132a96cd5 100644
@@ -1762,15 +1762,14 @@ static int set_bit_to_user(int nr, void __user *addr)
 	int bit = nr + (log % PAGE_SIZE) * 8;
 	int r;
 
-	r = get_user_pages_fast(log, 1, FOLL_WRITE, &page);
+	r = pin_user_pages_fast(log, 1, FOLL_WRITE, &page);
 	if (r < 0)
 		return r;
 	BUG_ON(r != 1);
 	base = kmap_atomic(page);
 	set_bit(bit, base);
 	kunmap_atomic(base);
-	set_page_dirty_lock(page);
-	put_page(page);
+	unpin_user_pages_dirty_lock(&page, 1, true);
 	return 0;
 }
 
Signature · 1 line
2.26.2