add callbacks for inaccessible pages
Subsystems (7)
v2 cover add callbacks for inaccessible pages
This patchset has a fixup for gup/mm, and provides the necessary arch hooks to enable protected virtualization.
Andrew: please simply squash/fixup the first patch into the appropriate one that is already in your tree.
v1 -> v2:
* use put_compound_head in the first patch
* fix commit message of the second patch
* minor code cleanups
* some comments to explain why sometimes we are not doing things
Claudio Imbrenda (2):
mm/gup: fixup for 9947ea2c1e608e32 "mm/gup: track FOLL_PIN pages" mm/gup/writeback: add callbacks for inaccessible pages
include/linux/gfp.h | 6 ++++ mm/gup.c | 73 +++++++++++++++++++++++++++++---------------- mm/page-writeback.c | 5 ++++ 3 files changed, 58 insertions(+), 26 deletions(-)
Signature · 1 line
2.24.1
PATCH v2 1/2 mm/gup: fixup for 9947ea2c1e608e32 "mm/gup: track FOLL_PIN pages"
In case pin fails, we need to unpin, a simple put_page will not be enough
fixup for commit 9947ea2c1e608e32 ("mm/gup: track FOLL_PIN pages")
it can be simply squashed in
--- mm/gup.c | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-)
mm/gup.c +23 -23
@@ -116,6 +116,28 @@ static __maybe_unused struct page *try_grab_compound_head(struct page *page,return NULL ; } +static void put_compound_head (struct page *page,int refs,unsigned int flags) +{ +if (flags &FOLL_PIN ) { +mod_node_page_state (page_pgdat (page),NR_FOLL_PIN_RELEASED , + refs); + +if (hpage_pincount_available (page)) +hpage_pincount_sub (page, refs); +else + refs *=GUP_PIN_COUNTING_BIAS ; + } + +VM_BUG_ON_PAGE (page_ref_count (page) < refs, page); +/* +* Calling put_page() for each ref is unnecessarily slow. Only the last +* ref needs a put_page(). +*/ +if (refs >1 ) +page_ref_sub (page, refs -1 ); +put_page (page); +} +/** * try_grab_page() - elevate a page's refcount by a flag-dependent amount *
@@ -2134,7 +2156,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (unlikely (pte_val (pte) !=pte_val (*ptep))) { -put_page (head); +put_compound_head (head,1 , flags); goto pte_unmap; }
@@ -2267,28 +2289,6 @@ static int record_subpages(struct page *page, unsigned long addr,return nr; } -static void put_compound_head (struct page *page,int refs,unsigned int flags) -{ -if (flags &FOLL_PIN ) { -mod_node_page_state (page_pgdat (page),NR_FOLL_PIN_RELEASED , - refs); - -if (hpage_pincount_available (page)) -hpage_pincount_sub (page, refs); -else - refs *=GUP_PIN_COUNTING_BIAS ; - } - -VM_BUG_ON_PAGE (page_ref_count (page) < refs, page); -/* -* Calling put_page() for each ref is unnecessarily slow. Only the last -* ref needs a put_page(). -*/ -if (refs >1 ) -page_ref_sub (page, refs -1 ); -put_page (page); -} -#ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end (unsigned long addr,unsigned long end,unsigned long sz)
Signature · 1 line
2.24.1
John Hubbard
reviewerRe: [PATCH v2 1/2] mm/gup: fixup for 9947ea2c1e608e32 "mm/gup: track FOLL_PIN pages"
Quoted 11 lines · Claudio Imbrenda
In case pin fails, we need to unpin, a simple
put_pagewill not be enoughfixup for commit
9947ea2c1e608e32("mm/gup: trackFOLL_PINpages")it can be simply squashed in
Signed-off-by Claudio Imbrenda <[email protected]>--- mm/gup.c | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-)
Looks good, thanks for fixing this.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 72 lines
mm/gup.c +23
-23@@ -116,6 +116,28 @@ static __maybe_unused struct page *try_grab_compound_head(struct page *page,return NULL ; } +static void put_compound_head (struct page *page,int refs,unsigned int flags) +{ +if (flags &FOLL_PIN ) { +mod_node_page_state (page_pgdat (page),NR_FOLL_PIN_RELEASED , + refs); + +if (hpage_pincount_available (page)) +hpage_pincount_sub (page, refs); +else + refs *=GUP_PIN_COUNTING_BIAS ; + } + +VM_BUG_ON_PAGE (page_ref_count (page) < refs, page); +/* +* Calling put_page() for each ref is unnecessarily slow. Only the last +* ref needs a put_page(). +*/ +if (refs >1 ) +page_ref_sub (page, refs -1 ); +put_page (page); +} +/** * try_grab_page() - elevate a page's refcount by a flag-dependent amount * @@ -2134,7 +2156,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (unlikely (pte_val (pte) !=pte_val (*ptep))) { -put_page (head); +put_compound_head (head,1 , flags); goto pte_unmap; }@@ -2267,28 +2289,6 @@ static int record_subpages(struct page *page, unsigned long addr,return nr; } -static void put_compound_head (struct page *page,int refs,unsigned int flags) -{ -if (flags &FOLL_PIN ) { -mod_node_page_state (page_pgdat (page),NR_FOLL_PIN_RELEASED , - refs); - -if (hpage_pincount_available (page)) -hpage_pincount_sub (page, refs); -else - refs *=GUP_PIN_COUNTING_BIAS ; - } - -VM_BUG_ON_PAGE (page_ref_count (page) < refs, page); -/* -* Calling put_page() for each ref is unnecessarily slow. Only the last -* ref needs a put_page(). -*/ -if (refs >1 ) -page_ref_sub (page, refs -1 ); -put_page (page); -} -#ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end (unsigned long addr,unsigned long end,unsigned long sz)
PATCH v2 2/2 mm/gup/writeback: add callbacks for inaccessible pages
In reply to Claudio ImbrendaWith the introduction of protected KVM guests on s390 there is now a concept of inaccessible pages. These pages need to be made accessible before the host can access them.
While cpu accesses will trigger a fault that can be resolved, I/O accesses will just fail. We need to add a callback into architecture code for places that will do I/O, namely when writeback is started or when a page reference is taken.
This is not only to enable paging, file backing etc, it is also necessary to protect the host against a malicious user space. For example a bad QEMU could simply start direct I/O on such protected memory. We do not want userspace to be able to trigger I/O errors and thus the logic is "whenever somebody accesses that page (gup) or does I/O, make sure that this page can be accessed". When the guest tries to access that page we will wait in the page fault handler for writeback to have finished and for the page_ref to be the expected value.
On s390x the function is not supposed to fail, so it is ok to use a WARN_ON on failure. If we ever need some more finegrained handling we can tackle this when we know the details.
--- include/linux/gfp.h | 6 ++++++ mm/gup.c | 27 ++++++++++++++++++++++++--- mm/page-writeback.c | 5 +++++ 3 files changed, 35 insertions(+), 3 deletions(-)
include/linux/gfp.h +6 -0
@@ -485,6 +485,12 @@ static inline void arch_free_page(struct page *page, int order) { }#ifndef HAVE_ARCH_ALLOC_PAGE static inline void arch_alloc_page (struct page *page,int order) { }#endif +#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE +static inline int arch_make_page_accessible (struct page *page) +{ +return 0 ; +} +#endif struct page * __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
mm/gup.c +24 -3
@@ -413,6 +413,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,struct page *page;spinlock_t *ptl;pte_t *ptep, pte; +int ret;/* FOLL_GET and FOLL_PIN are mutually exclusive. */ if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==
@@ -471,8 +472,6 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,if (is_zero_pfn (pte_pfn (pte))) { page =pte_page (pte); }else { -int ret; - ret =follow_pfn_pte (vma, address, ptep, flags); page =ERR_PTR (ret); goto out;
@@ -480,7 +479,6 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, }if (flags &FOLL_SPLIT &&PageTransCompound (page)) { -int ret;get_page (page);pte_unmap_unlock (ptep, ptl);lock_page (page);
@@ -497,6 +495,19 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, page =ERR_PTR (-ENOMEM ); goto out; } +/* +* We need to make the page accessible if we are actually going to +* poke at its content (pin), otherwise we can leave it inaccessible. +* If we cannot make the page accessible, fail. +*/ +if (flags &FOLL_PIN ) { + ret =arch_make_page_accessible (page); +if (ret) { +unpin_user_page (page); + page =ERR_PTR (ret); + goto out; + } + }if (flags &FOLL_TOUCH ) {if ((flags &FOLL_WRITE ) && !pte_dirty (pte) && !PageDirty (page))
@@ -2162,6 +2173,16 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,VM_BUG_ON_PAGE (compound_head (page) != head, page); +/* +* We need to make the page accessible if we are actually +* going to poke at its content (pin), otherwise we can +* leave it inaccessible. If the page cannot be made +* accessible, fail. +*/ +if ((flags &FOLL_PIN ) &&arch_make_page_accessible (page)) { +unpin_user_page (page); + goto pte_unmap; + }SetPageReferenced (page); pages[*nr] = page; (*nr)++;
mm/page-writeback.c +5 -0
@@ -2807,6 +2807,11 @@ int __test_set_page_writeback(struct page *page, bool keep_write)inc_zone_page_state (page,NR_ZONE_WRITE_PENDING ); }unlock_page_memcg (page); +/* +* If writeback has been triggered on a page that cannot be made +* accessible, it is too late. +*/ +WARN_ON (arch_make_page_accessible (page));return ret; }
Signature · 1 line
2.24.1
John Hubbard
reviewerRe: [PATCH v2 2/2] mm/gup/writeback: add callbacks for inaccessible pages
Quoted 86 lines · Claudio Imbrenda
With the introduction of protected KVM guests on s390 there is now a concept of inaccessible pages. These pages need to be made accessible before the host can access them.
While cpu accesses will trigger a fault that can be resolved, I/O accesses will just fail. We need to add a callback into architecture code for places that will do I/O, namely when writeback is started or when a page reference is taken.
This is not only to enable paging, file backing etc, it is also necessary to protect the host against a malicious user space. For example a bad QEMU could simply start direct I/O on such protected memory. We do not want userspace to be able to trigger I/O errors and thus the logic is "whenever somebody accesses that page (gup) or does I/O, make sure that this page can be accessed". When the guest tries to access that page we will wait in the page fault handler for writeback to have finished and for the
page_refto be the expected value.On s390x the function is not supposed to fail, so it is ok to use a
WARN_ONon failure. If we ever need some more finegrained handling we can tackle this when we know the details.Signed-off-by Claudio Imbrenda <[email protected]>Acked-by Will Deacon <[email protected]>Reviewed-by David Hildenbrand <[email protected]>Reviewed-by Christian Borntraeger <[email protected]>--- include/linux/gfp.h | 6 ++++++ mm/gup.c | 27 ++++++++++++++++++++++++--- mm/page-writeback.c | 5 +++++ 3 files changed, 35 insertions(+), 3 deletions(-)include/linux/gfp.h +6
-0@@ -485,6 +485,12 @@ static inline void arch_free_page(struct page *page, int order) { }#ifndef HAVE_ARCH_ALLOC_PAGE static inline void arch_alloc_page (struct page *page,int order) { }#endif +#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE +static inline int arch_make_page_accessible (struct page *page) +{ +return 0 ; +} +#endif struct page * __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,mm/gup.c +4
-3@@ -413,6 +413,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,struct page *page;spinlock_t *ptl;pte_t *ptep, pte; +int ret;/* FOLL_GET and FOLL_PIN are mutually exclusive. */ if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==@@ -471,8 +472,6 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,if (is_zero_pfn (pte_pfn (pte))) { page =pte_page (pte); }else { -int ret; - ret =follow_pfn_pte (vma, address, ptep, flags); page =ERR_PTR (ret); goto out;@@ -480,7 +479,6 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, }if (flags &FOLL_SPLIT &&PageTransCompound (page)) { -int ret;get_page (page);pte_unmap_unlock (ptep, ptl);lock_page (page);@@ -497,6 +495,19 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, page =ERR_PTR (-ENOMEM ); goto out; } +/* +* We need to make the page accessible if we are actually going to +* poke at its content (pin), otherwise we can leave it inaccessible.
mm/gup.c static struct page *follow_page_pte(struct vm_area_struct *vma, · line 510
Quoted 10 lines
mm/gup.c +10
-0+* If we cannot make the page accessible, fail. +*/ +if (flags &FOLL_PIN ) { + ret =arch_make_page_accessible (page); +if (ret) { +unpin_user_page (page); + page =ERR_PTR (ret); + goto out; + } + }
That looks good.
mm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2181
Quoted 13 lines
mm/gup.c +6
-0if (flags &FOLL_TOUCH ) {if ((flags &FOLL_WRITE ) && !pte_dirty (pte) && !PageDirty (page))@@ -2162,6 +2173,16 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,VM_BUG_ON_PAGE (compound_head (page) != head, page); +/* +* We need to make the page accessible if we are actually +* going to poke at its content (pin), otherwise we can +* leave it inaccessible. If the page cannot be made +* accessible, fail. +*/
This part looks good, so these two points are just nits:
That's a little bit of repeating what the code does, in the comments. How about:
/*
* We need to make the page accessible if and only if we are
* going to access its content (the FOLL_PIN case). Please see
* Documentation/core-api/pin_user_pages.rst for details. */
mm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2185
Quoted 4 lines
mm/gup.c +4
-0+if ((flags &FOLL_PIN ) &&arch_make_page_accessible (page)) { +unpin_user_page (page); + goto pte_unmap; + }
Your style earlier in the patch was easier on the reader, why not stay consistent with that (and with this file, which tends also to do this), so:
if (flags & FOLL_PIN) {
ret = arch_make_page_accessible(page);
if (ret) {
unpin_user_page(page);
goto pte_unmap;
}
}Quoted 8 lines
mm/gup.c +0
-0SetPageReferenced (page); pages[*nr] = page; (*nr)++;mm/page-writeback.c +0
-0
mm/page-writeback.c int __test_set_page_writeback(struct page *page, bool keep_write) · line 2814
Quoted 8 lines
mm/page-writeback.c +5
-0@@ -2807,6 +2807,11 @@ int __test_set_page_writeback(struct page *page, bool keep_write)inc_zone_page_state (page,NR_ZONE_WRITE_PENDING ); }unlock_page_memcg (page); +/* +* If writeback has been triggered on a page that cannot be made +* accessible, it is too late. +*/ +WARN_ON (arch_make_page_accessible (page));
I'm not deep enough into this area to know if a) this is correct, and b) if there are any other places that need arch_make_page_accessible() calls. So I'll rely on other reviewers to help check on that.
mm/page-writeback.c int __test_set_page_writeback(struct page *page, bool keep_write) · line 2817
Quoted 4 lines
mm/page-writeback.c +0
-0return ret; }
Anyway, I don't see any problems, and as I said, those documentation and style points are just nitpicks, not bugs.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [PATCH v2 2/2] mm/gup/writeback: add callbacks for inaccessible pages
Quoted 4 lines
Your style earlier in the patch was easier on the reader, why not stay consistent with that (and with this file, which tends also to do this), so:
Quoted 8 lines
if (flags & FOLL_PIN) {
ret = arch_make_page_accessible(page);
if (ret) {
unpin_user_page(page);
goto pte_unmap;
}
}+1
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 2/2] mm/gup/writeback: add callbacks for inaccessible pages
In reply to John HubbardQuoted 183 lines · On Mon, 2 Mar 2020 23:59:32 -0800 John Hubbard <[email protected]> wrote:
Quoted 88 lines · Claudio Imbrenda
With the introduction of protected KVM guests on s390 there is now a concept of inaccessible pages. These pages need to be made accessible before the host can access them.
While cpu accesses will trigger a fault that can be resolved, I/O accesses will just fail. We need to add a callback into architecture code for places that will do I/O, namely when writeback is started or when a page reference is taken.
This is not only to enable paging, file backing etc, it is also necessary to protect the host against a malicious user space. For example a bad QEMU could simply start direct I/O on such protected memory. We do not want userspace to be able to trigger I/O errors and thus the logic is "whenever somebody accesses that page (gup) or does I/O, make sure that this page can be accessed". When the guest tries to access that page we will wait in the page fault handler for writeback to have finished and for the
page_refto be the expected value.On s390x the function is not supposed to fail, so it is ok to use a
WARN_ONon failure. If we ever need some more finegrained handling we can tackle this when we know the details.Signed-off-by Claudio Imbrenda <[email protected]>Acked-by Will Deacon <[email protected]>Reviewed-by David Hildenbrand <[email protected]>Reviewed-by Christian Borntraeger <[email protected]>--- include/linux/gfp.h | 6 ++++++ mm/gup.c | 27 ++++++++++++++++++++++++--- mm/page-writeback.c | 5 +++++ 3 files changed, 35 insertions(+), 3 deletions(-)include/linux/gfp.h +0
-0@@ -485,6 +485,12 @@ static inline void arch_free_page(struct page *page, int order) { } #ifndef HAVE_ARCH_ALLOC_PAGEstatic inline void arch_alloc_page (struct page *page,int order){ } #endif
include/linux/gfp.h +6
-0+#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE +static inline int arch_make_page_accessible (struct page *page) +{ +return 0 ; +} +#endif struct page * __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int
preferred_nid, diff --gita/mm/gup.cb/mm/gup.c
index81a95fbe9901..15c47e0e86f8100644mm/gup.c +3
-3@@ -413,6 +413,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, struct page *page;spinlock_t *ptl;pte_t *ptep, pte; +int ret;/* FOLL_GET and FOLL_PIN are mutually exclusive. */ if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==@@ -471,8 +472,6 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, if (is_zero_pfn(pte_pfn(pte))) { page =pte_page (pte); } else { -int ret; - ret =follow_pfn_pte (vma, address, ptep, flags); page =ERR_PTR (ret); goto out;@@ -480,7 +479,6 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, }if (flags &FOLL_SPLIT &&PageTransCompound (page)) { -int ret;get_page (page);pte_unmap_unlock (ptep, ptl);lock_page (page);@@ -497,6 +495,19 @@ static struct page *follow_page_pte(structvm_area_struct *vma, page =ERR_PTR (-ENOMEM ); goto out; } +/* +* We need to make the page accessible if we are actually going to
mm/gup.c +1
-0+* poke at its content (pin), otherwise we can leave it inaccessible.
mm/gup.c static struct page *follow_page_pte(struct vm_area_struct *vma, · line 510
Quoted 10 lines
mm/gup.c +10
-0+* If we cannot make the page accessible, fail. +*/ +if (flags &FOLL_PIN ) { + ret =arch_make_page_accessible (page); +if (ret) { +unpin_user_page (page); + page =ERR_PTR (ret); + goto out; + } + }That looks good.
Quoted 14 lines
if (flags & FOLL_TOUCH) { if ((flags & FOLL_WRITE) && !pte_dirty(pte) && !PageDirty(page))mm/gup.c +2
-0@@ -2162,6 +2173,16 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, VM_BUG_ON_PAGE(compound_head(page) != head, page); +/* +* We need to make the page accessible if we are actually
mm/gup.c +1
-0+* going to poke at its content (pin), otherwise we can
mm/gup.c +1
-0+* leave it inaccessible. If the page cannot be made
mm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2181
Quoted 2 lines
mm/gup.c +2
-0+* accessible, fail. +*/ This part looks good, so these two points are just nits:
That's a little bit of repeating what the code does, in the comments. How about:
/* * We need to make the page accessible if and only ifwe are
* going to access its content (theFOLL_PINcase).
Please see
*Documentation/core-api/pin_user_pages.rstfor
details. */mm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2185
Quoted 5 lines
mm/gup.c +1
-0+ if ((flags & FOLL_PIN) &&
arch_make_page_accessible(page)) {mm/gup.c +3
-0+unpin_user_page (page); + goto pte_unmap; + }Your style earlier in the patch was easier on the reader, why not stay consistent with that (and with this file, which tends also to do this), so:
if (flags & FOLL_PIN) { ret = arch_make_page_accessible(page); if (ret) { unpin_user_page(page); goto pte_unmap; } }Quoted 14 lines
SetPageReferenced(page); pages[*nr] = page; (*nr)++;mm/page-writeback.c +2
-0@@ -2807,6 +2807,11 @@ int __test_set_page_writeback(struct page *page, bool keep_write) inc_zone_page_state(page,NR_ZONE_WRITE_PENDING ); }unlock_page_memcg (page); +/* +* If writeback has been triggered on a page that cannot be made
mm/page-writeback.c int __test_set_page_writeback(struct page *page, bool keep_write) · line 2814
Quoted 3 lines
mm/page-writeback.c +3
-0+* accessible, it is too late. +*/ +WARN_ON (arch_make_page_accessible (page));I'm not deep enough into this area to know if a) this is correct, and
b) if there are any other places that needarch_make_page_accessible()calls. So I'll rely on other reviewers to help check on that.
Quoted 7 lines
Quoted 4 lines
return ret;
}Anyway, I don't see any problems, and as I said, those documentation and style points are just nitpicks, not bugs.
these are minor fixes, and I mostly agree with you. I'll fix them and send a v3 soon™
thanks for the comments!