KVM: Restricted mapping of guest_memfd at the host and arm64 support
Subsystems (9)
v5 cover KVM: Restricted mapping of guest_memfd at the host and arm64 support
This series adds restricted mmap() support to guest_memfd, as well as support for guest_memfd on arm64. It is based on Linux 6.13-rc7. Please refer to v3 for the context [1].
Main changes since v4 [2]:
- Fixed handling of guest_memfd()-backed page faults in arm64
- Rebased on Linux 6.13-rc7
Not a change per se, but I am able to trigger/test the callback on the final __folio_put() using vmsplice to grab a reference without increasing the mapcount.
The state diagram that uses the new states in this patch series, and how they would interact with sharing/unsharing in pKVM [3].
Cheers,
/fuad
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lpc.events/event/18/contributions/1758/attachments/1457/3699/Guestmemfd%20folio%20state%20page_type.pdf
Ackerley Tng (2):
KVM: guest_memfd: Make guest mem use guest mem inodes instead of
anonymous inodes
KVM: guest_memfd: Track mappability within a struct kvm_gmem_private
Fuad Tabba (13):
mm: Consolidate freeing of typed folios on final folio_put()
KVM: guest_memfd: Introduce kvm_gmem_get_pfn_locked(), which retains
the folio lock
KVM: guest_memfd: Folio mappability states and functions that manage
their transition
KVM: guest_memfd: Handle final folio_put() of guestmem pages
KVM: guest_memfd: Allow host to mmap guest_memfd() pages when shared
KVM: guest_memfd: Add guest_memfd support to
kvm_(read|/write)_guest_page()
KVM: guest_memfd: Add KVM capability to check if guest_memfd is host
mappable
KVM: guest_memfd: Add a guest_memfd() flag to initialize it as
mappable
KVM: guest_memfd: selftests: guest_memfd mmap() test when mapping is
allowed
KVM: arm64: Skip VMA checks for slots without userspace address
KVM: arm64: Refactor user_mem_abort() calculation of force_pte
KVM: arm64: Handle guest_memfd()-backed guest page faults
KVM: arm64: Enable guest_memfd private memory when pKVM is enabled
Documentation/virt/kvm/api.rst | 4 + arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/mmu.c | 98 ++- include/linux/kvm_host.h | 80 +++ include/linux/page-flags.h | 22 + include/uapi/linux/kvm.h | 2 + include/uapi/linux/magic.h | 1 + mm/debug.c | 1 + mm/swap.c | 28 +- tools/testing/selftests/kvm/Makefile | 1 + .../testing/selftests/kvm/guest_memfd_test.c | 64 +- virt/kvm/Kconfig | 4 + virt/kvm/guest_memfd.c | 579 +++++++++++++++++- virt/kvm/kvm_main.c | 234 ++++++- 15 files changed, 1034 insertions(+), 88 deletions(-)
base-commit: 5bc55a333a2f7316b58edc7573e8e893f7acb532
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 1/15 mm: Consolidate freeing of typed folios on final folio_put()
Some folio types, such as hugetlb, handle freeing their own folios. Moreover, guest_memfd will require being notified once a folio's reference count reaches 0 to facilitate shared to private folio conversion, without the folio actually being freed at that point.
As a first step towards that, this patch consolidates freeing folios that have a type. The first user is hugetlb folios. Later in this patch series, guest_memfd will become the second user of this.
--- include/linux/page-flags.h | 15 +++++++++++++++ mm/swap.c | 24 +++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)
include/linux/page-flags.h +15 -0
@@ -962,6 +962,21 @@ static inline bool page_has_type(const struct page *page)return page_mapcount_is_type (data_race (page->page_type)); } +static inline int page_get_type (const struct page *page) +{ +return page->page_type >>24 ; +} + +static inline bool folio_has_type (const struct folio *folio) +{ +return page_has_type (&folio->page); +} + +static inline int folio_get_type (const struct folio *folio) +{ +return page_get_type (&folio->page); +} +#define FOLIO_TYPE_OPS (lname, fname) \ static __always_inline bool folio_test_##fname(const struct folio *folio) \ { \
mm/swap.c +19 -5
@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); } +static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) { +case PGTY_hugetlb: +free_huge_folio (folio); +return ; +case PGTY_offline: +/* Nothing to do, it's offline. */ +return ; +default : +WARN_ON_ONCE (1 ); + } +} +void __folio_put (struct folio *folio) {if (unlikely (folio_is_zone_device (folio))) {
@@ -101,8 +115,8 @@ void __folio_put(struct folio *folio)return ; } -if (folio_test_hugetlb (folio)) { -free_huge_folio (folio); +if (unlikely (folio_has_type (folio))) { +free_typed_folio (folio);return ; }
@@ -934,13 +948,13 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)if (!folio_ref_sub_and_test (folio, nr_refs))continue ; -/* hugetlb has its own memcg */ -if (folio_test_hugetlb (folio)) { +if (unlikely (folio_has_type (folio))) { +/* typed folios have their own memcg, if any */ if (lruvec) {unlock_page_lruvec_irqrestore (lruvec, flags); lruvec =NULL ; } -free_huge_folio (folio); +free_typed_folio (folio);continue ; }folio_unqueue_deferred_split (folio);
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
Quoted 49 lines · Fuad Tabba
Some folio types, such as hugetlb, handle freeing their own folios. Moreover,
guest_memfdwill require being notified once a folio's reference count reaches 0 to facilitate shared to private folio conversion, without the folio actually being freed at that point.As a first step towards that, this patch consolidates freeing folios that have a type. The first user is hugetlb folios. Later in this patch series,
guest_memfdwill become the second user of this.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/linux/page-flags.h | 15 +++++++++++++++ mm/swap.c | 24 +++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)include/linux/page-flags.h +15
-0@@ -962,6 +962,21 @@ static inline bool page_has_type(const struct page *page)return page_mapcount_is_type (data_race (page->page_type)); } +static inline int page_get_type (const struct page *page) +{ +return page->page_type >>24 ; +} + +static inline bool folio_has_type (const struct folio *folio) +{ +return page_has_type (&folio->page); +} + +static inline int folio_get_type (const struct folio *folio) +{ +return page_get_type (&folio->page); +} +#define FOLIO_TYPE_OPS (lname, fname) \ static __always_inline bool folio_test_##fname(const struct folio *folio) \ { \mm/swap.c +0
-0
mm/swap.c static void page_cache_release(struct folio *folio) · line 99
Quoted 6 lines
mm/swap.c +3
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); } +static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) {
I think you need:
+#if IS_ENABLED(CONFIG_HUGETLBFS)
mm/swap.c static void page_cache_release(struct folio *folio) · line 102
Quoted 3 lines
mm/swap.c +3
-0+case PGTY_hugetlb: +free_huge_folio (folio); +return ;
+#endif
I think this worked before because folio_test_hugetlb was defined by:FOLIO_TEST_FLAG_FALSE(hugetlb)
and evidently compiler optimizes out the free_huge_folio(folio) before linking.
You'll probably want to do the same for the PGTY_guestmem in the later patch!
Quoted 42 lines
mm/swap.c +13
-5+case PGTY_offline: +/* Nothing to do, it's offline. */ +return ; +default : +WARN_ON_ONCE (1 ); + } +} +void __folio_put (struct folio *folio) {if (unlikely (folio_is_zone_device (folio))) {@@ -101,8 +115,8 @@ void __folio_put(struct folio *folio)return ; } -if (folio_test_hugetlb (folio)) { -free_huge_folio (folio); +if (unlikely (folio_has_type (folio))) { +free_typed_folio (folio);return ; }@@ -934,13 +948,13 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)if (!folio_ref_sub_and_test (folio, nr_refs))continue ; -/* hugetlb has its own memcg */ -if (folio_test_hugetlb (folio)) { +if (unlikely (folio_has_type (folio))) { +/* typed folios have their own memcg, if any */ if (lruvec) {unlock_page_lruvec_irqrestore (lruvec, flags); lruvec =NULL ; } -free_huge_folio (folio); +free_typed_folio (folio);continue ; }folio_unqueue_deferred_split (folio);Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
Quoted 66 lines · Elliot Berman
Quoted 49 lines · Fuad Tabba
Some folio types, such as hugetlb, handle freeing their own folios. Moreover,
guest_memfdwill require being notified once a folio's reference count reaches 0 to facilitate shared to private folio conversion, without the folio actually being freed at that point.As a first step towards that, this patch consolidates freeing folios that have a type. The first user is hugetlb folios. Later in this patch series,
guest_memfdwill become the second user of this.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/linux/page-flags.h | 15 +++++++++++++++ mm/swap.c | 24 +++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)include/linux/page-flags.h +15
-0@@ -962,6 +962,21 @@ static inline bool page_has_type(const struct page *page)return page_mapcount_is_type (data_race (page->page_type)); } +static inline int page_get_type (const struct page *page) +{ +return page->page_type >>24 ; +} + +static inline bool folio_has_type (const struct folio *folio) +{ +return page_has_type (&folio->page); +} + +static inline int folio_get_type (const struct folio *folio) +{ +return page_get_type (&folio->page); +} +#define FOLIO_TYPE_OPS (lname, fname) \ static __always_inline bool folio_test_##fname(const struct folio *folio) \ { \mm/swap.c +0
-0mm/swap.c static void page_cache_release(struct folio *folio) · line 99
Quoted 6 lines
mm/swap.c +3
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); } +static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) {I think you need:
+#if
IS_ENABLED(CONFIG_HUGETLBFS)mm/swap.c static void page_cache_release(struct folio *folio) · line 102
Quoted 3 lines
mm/swap.c +3
-0+case PGTY_hugetlb: +free_huge_folio (folio); +return ;+#endif
Quoted 7 lines
I think this worked before because
folio_test_hugetlbwas defined by:FOLIO_TEST_FLAG_FALSE(hugetlb)
and evidently compiler optimizes out thefree_huge_folio(folio)before linking.You'll probably want to do the same for the
PGTY_guestmemin the later patch!
Thanks Elliot. This will keep the kernel test robot happy when I respin.
Cheers,
/fuad
Quoted 43 lines
Quoted 43 lines
mm/swap.c +13
-6+case PGTY_offline: +/* Nothing to do, it's offline. */ +return ; +default : +WARN_ON_ONCE (1 ); + } +} +void __folio_put (struct folio *folio) {if (unlikely (folio_is_zone_device (folio))) {@@ -101,8 +115,8 @@ void __folio_put(struct folio *folio)return ; } -if (folio_test_hugetlb (folio)) { -free_huge_folio (folio); +if (unlikely (folio_has_type (folio))) { +free_typed_folio (folio);return ; }@@ -934,13 +948,13 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)if (!folio_ref_sub_and_test (folio, nr_refs))continue ; -/* hugetlb has its own memcg */ -if (folio_test_hugetlb (folio)) { +if (unlikely (folio_has_type (folio))) { +/* typed folios have their own memcg, if any */ if (lruvec) {unlock_page_lruvec_irqrestore (lruvec, flags); lruvec =NULL ; } -free_huge_folio (folio); +free_typed_folio (folio);continue ; }folio_unqueue_deferred_split (folio); --2.48.0.rc2.279.g1de40edade-goog
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
In reply to Elliot BermanQuoted 65 lines · Elliot Berman
Quoted 49 lines · Fuad Tabba
Some folio types, such as hugetlb, handle freeing their own folios. Moreover,
guest_memfdwill require being notified once a folio's reference count reaches 0 to facilitate shared to private folio conversion, without the folio actually being freed at that point.As a first step towards that, this patch consolidates freeing folios that have a type. The first user is hugetlb folios. Later in this patch series,
guest_memfdwill become the second user of this.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/linux/page-flags.h | 15 +++++++++++++++ mm/swap.c | 24 +++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)include/linux/page-flags.h +15
-0@@ -962,6 +962,21 @@ static inline bool page_has_type(const struct page *page)return page_mapcount_is_type (data_race (page->page_type)); } +static inline int page_get_type (const struct page *page) +{ +return page->page_type >>24 ; +} + +static inline bool folio_has_type (const struct folio *folio) +{ +return page_has_type (&folio->page); +} + +static inline int folio_get_type (const struct folio *folio) +{ +return page_get_type (&folio->page); +} +#define FOLIO_TYPE_OPS (lname, fname) \ static __always_inline bool folio_test_##fname(const struct folio *folio) \ { \mm/swap.c +0
-0mm/swap.c static void page_cache_release(struct folio *folio) · line 99
Quoted 6 lines
mm/swap.c +3
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); } +static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) {I think you need:
+#if
IS_ENABLED(CONFIG_HUGETLBFS)mm/swap.c static void page_cache_release(struct folio *folio) · line 102
Quoted 3 lines
mm/swap.c +3
-0+case PGTY_hugetlb: +free_huge_folio (folio); +return ;+#endif
Quoted 4 lines
I think this worked before because
folio_test_hugetlbwas defined by:FOLIO_TEST_FLAG_FALSE(hugetlb)
and evidently compiler optimizes out thefree_huge_folio(folio)before linking.
Likely, we should be using
case PGTY_hugetlb: if(IF_ENABLED(CONFIG_HUGETLBFS)) free_huge_folio(folio); return:
if possible (I assume so).
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
Quoted 72 lines · David Hildenbrand
Quoted 63 lines · Elliot Berman
Quoted 49 lines · Fuad Tabba
Some folio types, such as hugetlb, handle freeing their own folios. Moreover,
guest_memfdwill require being notified once a folio's reference count reaches 0 to facilitate shared to private folio conversion, without the folio actually being freed at that point.As a first step towards that, this patch consolidates freeing folios that have a type. The first user is hugetlb folios. Later in this patch series,
guest_memfdwill become the second user of this.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/linux/page-flags.h | 15 +++++++++++++++ mm/swap.c | 24 +++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)include/linux/page-flags.h +15
-0@@ -962,6 +962,21 @@ static inline bool page_has_type(const struct page *page)return page_mapcount_is_type (data_race (page->page_type)); } +static inline int page_get_type (const struct page *page) +{ +return page->page_type >>24 ; +} + +static inline bool folio_has_type (const struct folio *folio) +{ +return page_has_type (&folio->page); +} + +static inline int folio_get_type (const struct folio *folio) +{ +return page_get_type (&folio->page); +} +#define FOLIO_TYPE_OPS (lname, fname) \ static __always_inline bool folio_test_##fname(const struct folio *folio) \ { \mm/swap.c +0
-0mm/swap.c static void page_cache_release(struct folio *folio) · line 99
Quoted 6 lines
mm/swap.c +3
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); } +static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) {I think you need:
+#if
IS_ENABLED(CONFIG_HUGETLBFS)mm/swap.c static void page_cache_release(struct folio *folio) · line 102
Quoted 3 lines
mm/swap.c +3
-0+case PGTY_hugetlb: +free_huge_folio (folio); +return ;Quoted 6 lines
+#endif
I think this worked before because
folio_test_hugetlbwas defined by:FOLIO_TEST_FLAG_FALSE(hugetlb)
and evidently compiler optimizes out thefree_huge_folio(folio)before linking.
Quoted 8 lines
Likely, we should be using
case PGTY_hugetlb: if(IF_ENABLED(CONFIG_HUGETLBFS)) free_huge_folio(folio); return:if possible (I assume so).
Yes it does. I'll fix it.
Cheers,
/fuad
Quoted 5 lines
--
Cheers,David / dhildenb
kernel test robot
build botRe: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
In reply to Fuad Tabbakernel test robot noticed the following build errors:
Hi Fuad,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on 5bc55a333a2f7316b58edc7573e8e893f7acb532]
url: https://github.com/intel-lab-lkp/linux/commits/Fuad-Tabba/mm-Consolidate-freeing-of-typed-folios-on-final-folio_put/20250118-003538 base: 5bc55a333a2f7316b58edc7573e8e893f7acb532 patch link: https://lore.kernel.org/r/20250117163001.2326672-2-tabba%40google.com patch subject: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put() config: i386-buildonly-randconfig-002-20250118 (https://download.01.org/0day-ci/archive/20250118/[email protected]/config) compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250118/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> ld.lld: error: undefined symbol: free_huge_folio >>> referenced by swap.c:101 (mm/swap.c:101) >>> mm/swap.o:(__folio_put) in archive vmlinux.a >>> referenced by swap.c:101 (mm/swap.c:101) >>> mm/swap.o:(folios_put_refs) in archive vmlinux.a
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
In reply to Fuad Tabbakernel test robot noticed the following build errors:
Hi Fuad,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on 5bc55a333a2f7316b58edc7573e8e893f7acb532]
url: https://github.com/intel-lab-lkp/linux/commits/Fuad-Tabba/mm-Consolidate-freeing-of-typed-folios-on-final-folio_put/20250118-003538 base: 5bc55a333a2f7316b58edc7573e8e893f7acb532 patch link: https://lore.kernel.org/r/20250117163001.2326672-2-tabba%40google.com patch subject: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put() config: i386-buildonly-randconfig-004-20250118 (https://download.01.org/0day-ci/archive/20250118/[email protected]/config) compiler: gcc-11 (Debian 11.3.0-12) 11.3.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250118/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
ld: mm/swap.o: in function `free_typed_folio':
>> swap.c:(.text+0x4f5): undefined reference to `free_huge_folio'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
In reply to Fuad TabbaQuoted 51 lines · Fuad Tabba
Some folio types, such as hugetlb, handle freeing their own folios. Moreover,
guest_memfdwill require being notified once a folio's reference count reaches 0 to facilitate shared to private folio conversion, without the folio actually being freed at that point.As a first step towards that, this patch consolidates freeing folios that have a type. The first user is hugetlb folios. Later in this patch series,
guest_memfdwill become the second user of this.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/linux/page-flags.h | 15 +++++++++++++++ mm/swap.c | 24 +++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)include/linux/page-flags.h +15
-0@@ -962,6 +962,21 @@ static inline bool page_has_type(const struct page *page)return page_mapcount_is_type (data_race (page->page_type)); } +static inline int page_get_type (const struct page *page) +{ +return page->page_type >>24 ; +} + +static inline bool folio_has_type (const struct folio *folio) +{ +return page_has_type (&folio->page); +} + +static inline int folio_get_type (const struct folio *folio) +{ +return page_get_type (&folio->page); +} +#define FOLIO_TYPE_OPS (lname, fname) \ static __always_inline bool folio_test_##fname(const struct folio *folio) \ { \mm/swap.c +0
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); }
mm/swap.c static void page_cache_release(struct folio *folio) · line 105
Quoted 10 lines
mm/swap.c +9
-0+static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) { +case PGTY_hugetlb: +free_huge_folio (folio); +return ; +case PGTY_offline: +/* Nothing to do, it's offline. */ +return ;
Please drop the PGTY_offline part for now, it was rather to highlight what could be done.
But the real goal will be to not make offline pages use the refcount at all (frozen).
If we really want the temporary PGTY_offline change, it should be introduced separately.
Apart from that LGTM!
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
Quoted 22 lines · David Hildenbrand
Quoted 7 lines · Fuad Tabba
mm/swap.c +0
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); }mm/swap.c static void page_cache_release(struct folio *folio) · line 105
Quoted 10 lines
mm/swap.c +9
-0+static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) { +case PGTY_hugetlb: +free_huge_folio (folio); +return ; +case PGTY_offline: +/* Nothing to do, it's offline. */ +return ;Please drop the
PGTY_offlinepart for now, it was rather to highlight what could be done.
Quoted 7 lines
But the real goal will be to not make offline pages use the refcount at all (frozen).
If we really want the temporary
PGTY_offlinechange, it should be introduced separately.Apart from that LGTM!
I gues you mean the WARN_ON_ONCE(1) should be dropped from the default:
handler as well, right? IIUC offline pages are not not yet frozen so there will be warnings otherwise. And I haven't check if the other types are frozen (I know slab is, very recently :)
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
Quoted 29 lines · Vlastimil Babka
Quoted 28 lines · David Hildenbrand
Quoted 7 lines · Fuad Tabba
mm/swap.c +0
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); }mm/swap.c static void page_cache_release(struct folio *folio) · line 105
Quoted 10 lines
mm/swap.c +9
-0+static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) { +case PGTY_hugetlb: +free_huge_folio (folio); +return ; +case PGTY_offline: +/* Nothing to do, it's offline. */ +return ;Please drop the
PGTY_offlinepart for now, it was rather to highlight what could be done.But the real goal will be to not make offline pages use the refcount at all (frozen).
If we really want the temporary
PGTY_offlinechange, it should be introduced separately.
Quoted 6 lines
Quoted 1 line
Apart from that LGTM!
I gues you mean the
WARN_ON_ONCE(1)should be dropped from the default:
handler as well, right? IIUC offline pages are not not yet frozen so there will be warnings otherwise. And I haven't check if the other types are frozen (I know slab is, very recently :)
Oh and also free_typed_folio() would have to become bool and if it returns false, the normal freeing proceeds?
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
In reply to Vlastimil BabkaQuoted 26 lines · Vlastimil Babka
Quoted 25 lines · David Hildenbrand
Quoted 7 lines · Fuad Tabba
mm/swap.c +0
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); }mm/swap.c static void page_cache_release(struct folio *folio) · line 105
Quoted 10 lines
mm/swap.c +9
-0+static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) { +case PGTY_hugetlb: +free_huge_folio (folio); +return ; +case PGTY_offline: +/* Nothing to do, it's offline. */ +return ;Please drop the
PGTY_offlinepart for now, it was rather to highlight what could be done.But the real goal will be to not make offline pages use the refcount at all (frozen).
Quoted 8 lines
Quoted 4 lines
If we really want the temporary
PGTY_offlinechange, it should be introduced separately.Apart from that LGTM!
I gues you mean the
WARN_ON_ONCE(1)should be dropped from the default:
handler as well, right? IIUC offline pages are not not yet frozen so there will be warnings otherwise.
If we get offline pages here, it is unexpected and wrong. All users clear PG_offline before handing them back to the buddy.
There is one nasty race in virtio-mem code for handling memory offlining with PG_offline pages, which we haven't seen so far in practice. See virtio_mem_fake_offline_going_offline()->page_ref_dec_and_test() for the nasty details. It would only trigger with some weird speculative references.
The proper fix will be do leave the refcount frozen for them, so speculative refcount users will just fail.
If we want to tackle that before we can do that, we should do it in a separate patch (and not buried in this series).
And I haven't check if the other types are
Quoted 1 line
frozen (I know slab is, very recently :)
I think if we would have that, we would end up triggering the free_pages_prepare()->free_page_is_bad(), because page->_mapcount == folio->mapcount would not be -1 for typed folios.
Right now we don't expect folios of specific types to ever get freed (in the future, these won't be folios anymore at all -- only guest_memfd and hugetlb would be folios, that need special care to be handed back to their pool).
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v5 01/15] mm: Consolidate freeing of typed folios on final folio_put()
In reply to David HildenbrandQuoted 54 lines · David Hildenbrand
Quoted 52 lines · Fuad Tabba
Some folio types, such as hugetlb, handle freeing their own folios. Moreover,
guest_memfdwill require being notified once a folio's reference count reaches 0 to facilitate shared to private folio conversion, without the folio actually being freed at that point.As a first step towards that, this patch consolidates freeing folios that have a type. The first user is hugetlb folios. Later in this patch series,
guest_memfdwill become the second user of this.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/linux/page-flags.h | 15 +++++++++++++++ mm/swap.c | 24 +++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)include/linux/page-flags.h +15
-0@@ -962,6 +962,21 @@ static inline bool page_has_type(const struct page *page)return page_mapcount_is_type (data_race (page->page_type)); } +static inline int page_get_type (const struct page *page) +{ +return page->page_type >>24 ; +} + +static inline bool folio_has_type (const struct folio *folio) +{ +return page_has_type (&folio->page); +} + +static inline int folio_get_type (const struct folio *folio) +{ +return page_get_type (&folio->page); +} +#define FOLIO_TYPE_OPS (lname, fname) \ static __always_inline bool folio_test_##fname(const struct folio *folio) \ { \mm/swap.c +0
-0@@ -94,6 +94,20 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); }
Quoted 12 lines
mm/swap.c static void page_cache_release(struct folio *folio) · line 105
Quoted 9 lines
mm/swap.c +9
-0+static void free_typed_folio (struct folio *folio) +{ +switch (folio_get_type (folio)) { +case PGTY_hugetlb: +free_huge_folio (folio); +return ; +case PGTY_offline: +/* Nothing to do, it's offline. */ +return ;Please drop the
PGTY_offlinepart for now, it was rather to highlight what could be done.
Will do.
Thanks,
/fuad
Quoted 14 lines
But the real goal will be to not make offline pages use the refcount at all (frozen).
If we really want the temporary
PGTY_offlinechange, it should be introduced separately.Apart from that LGTM!
--
Cheers,David / dhildenb
RFC PATCH v5 2/15 KVM: guest_memfd: Make guest mem use guest mem inodes instead of anonymous inodes
In reply to Fuad TabbaFrom: Ackerley Tng <[email protected]>
Using guest mem inodes allows us to store metadata for the backing memory on the inode. Metadata will be added in a later patch to support HugeTLB pages.
Metadata about backing memory should not be stored on the file, since the file represents a guest_memfd's binding with a struct kvm, and metadata about backing memory is not unique to a specific binding and struct kvm.
--- include/uapi/linux/magic.h | 1 + virt/kvm/guest_memfd.c | 119 ++++++++++++++++++++++++++++++------- 2 files changed, 100 insertions(+), 20 deletions(-)
include/uapi/linux/magic.h +1 -0
@@ -103,5 +103,6 @@#define DEVMEM_MAGIC 0x454d444d/* "DMEM" */ #define SECRETMEM_MAGIC 0x5345434d/* "SECM" */ #define PID_FS_MAGIC 0x50494446/* "PIDF" */ +#define GUEST_MEMORY_MAGIC 0x474d454d/* "GMEM" */ #endif /* __LINUX_MAGIC_H__ */
virt/kvm/guest_memfd.c +99 -20
166 lines not on this page · raw message
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
Re: [RFC PATCH v5 02/15] KVM: guest_memfd: Make guest mem use guest mem inodes instead of anonymous inodes
Hi Fuad,
Quoted 34 lines · Fuad Tabba
From: Ackerley Tng <[email protected]>
Using guest mem inodes allows us to store metadata for the backing memory on the inode. Metadata will be added in a later patch to support HugeTLB pages.
Metadata about backing memory should not be stored on the file, since the file represents a guest_memfd's binding with a struct kvm, and metadata about backing memory is not unique to a specific binding and struct kvm.
Signed-off-by Ackerley Tng <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/uapi/linux/magic.h | 1 + virt/kvm/guest_memfd.c | 119 ++++++++++++++++++++++++++++++------- 2 files changed, 100 insertions(+), 20 deletions(-)include/uapi/linux/magic.h +1
-0@@ -103,5 +103,6 @@#define DEVMEM_MAGIC 0x454d444d/* "DMEM" */ #define SECRETMEM_MAGIC 0x5345434d/* "SECM" */ #define PID_FS_MAGIC 0x50494446/* "PIDF" */ +#define GUEST_MEMORY_MAGIC 0x474d454d/* "GMEM" */ #endif /* __LINUX_MAGIC_H__ */ virt/kvm/guest_memfd.c +0
-0
virt/kvm/guest_memfd.c line 3
Quoted 3 lines
virt/kvm/guest_memfd.c +2
-0@@ -1,12 +1,17 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/fs.h> +#include <linux/mount.h>
This can be dropped since "linux/mount.h" has been included to "linux/fs.h".
Quoted 96 lines
virt/kvm/guest_memfd.c +67
-0#include <linux/backing-dev.h> #include <linux/falloc.h> #include <linux/kvm_host.h> +#include <linux/pseudo_fs.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> #include "kvm_mm.h" +static struct vfsmount *kvm_gmem_mnt; +struct kvm_gmem {struct kvm *kvm;struct xarray bindings;@@ -307,6 +312,38 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static const struct super_operations kvm_gmem_super_operations = { + .statfs = simple_statfs, +}; + +static int kvm_gmem_init_fs_context (struct fs_context *fc) +{ +struct pseudo_fs_context *ctx; + +if (!init_pseudo (fc,GUEST_MEMORY_MAGIC )) +return -ENOMEM ; + + ctx = fc->fs_private; + ctx->ops = &kvm_gmem_super_operations; + +return 0 ; +} + +static struct file_system_type kvm_gmem_fs = { + .name ="kvm_guest_memory" , + .init_fs_context = kvm_gmem_init_fs_context, + .kill_sb = kill_anon_super, +}; + +static void kvm_gmem_init_mount (void ) +{ + kvm_gmem_mnt =kern_mount (&kvm_gmem_fs); +BUG_ON (IS_ERR (kvm_gmem_mnt)); + +/* For giggles. Userspace can never map this anyways. */ + kvm_gmem_mnt->mnt_flags |=MNT_NOEXEC ; +} +static struct file_operations kvm_gmem_fops = { .open = generic_file_open, .release = kvm_gmem_release,@@ -316,6 +353,8 @@ static struct file_operations kvm_gmem_fops = {void kvm_gmem_init (struct module *module) { kvm_gmem_fops.owner = module; + +kvm_gmem_init_mount (); } static int kvm_gmem_migrate_folio(struct address_space *mapping,@@ -397,11 +436,67 @@ static const struct inode_operations kvm_gmem_iops = { .setattr = kvm_gmem_setattr, }; +static struct inode *kvm_gmem_inode_make_secure_inode (const char *name, +loff_t size,u64 flags) +{ +const struct qstr qname =QSTR_INIT (name,strlen (name)); +struct inode *inode; +int err; + + inode =alloc_anon_inode (kvm_gmem_mnt->mnt_sb); +if (IS_ERR (inode)) +return inode; + + err =security_inode_init_security_anon (inode, &qname,NULL ); +if (err) { +iput (inode); +return ERR_PTR (err); + } + + inode->i_private = (void *)(unsigned long )flags; + inode->i_op = &kvm_gmem_iops; + inode->i_mapping->a_ops = &kvm_gmem_aops; + inode->i_mode |=S_IFREG ; + inode->i_size = size; +mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); +mapping_set_inaccessible (inode->i_mapping); +/* Unmovable mappings are supposed to be marked unevictable as well. */ +WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); + +return inode; +} +
virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 478
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+static struct file *kvm_gmem_inode_create_getfile (void *priv,loff_t size, +u64 flags) +{ +static const char *name ="[kvm-gmem]" ; +struct inode *inode; +struct file *file; + +if (kvm_gmem_fops.owner && !try_module_get (kvm_gmem_fops.owner)) +return ERR_PTR (-ENOENT ); +
The validation on 'kvm_gmem_fops.owner' can be removed since try_module_get() and module_put() are friendly to a NULL parameter, even when CONFIG_MODULE_UNLOAD == N
A module_put(kvm_gmem_fops.owner) is needed in the various erroneous cases in this function. Otherwise, the reference count of the owner (module) will become imbalanced on any errors.
virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 482
Quoted 4 lines
virt/kvm/guest_memfd.c +4
-0+ inode =kvm_gmem_inode_make_secure_inode (name, size, flags); +if (IS_ERR (inode)) +return ERR_CAST (inode); +
ERR_CAST may be dropped since there is nothing to be casted or converted?
virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 493
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ file =alloc_file_pseudo (inode, kvm_gmem_mnt, name,O_RDWR , + &kvm_gmem_fops); +if (IS_ERR (file)) { +iput (inode); +return file; + } + + file->f_mapping = inode->i_mapping; + file->f_flags |=O_LARGEFILE ; + file->private_data = priv; +
'file->f_mapping = inode->i_mapping' may be dropped since it's already correctly set by alloc_file_pseudo().
alloc_file_pseudo
alloc_path_pseudo
alloc_file
alloc_empty_file
file_init_path // Set by this function
Quoted 36 lines
virt/kvm/guest_memfd.c +4
-17+return file; +} +static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) { -const char *anon_name ="[kvm-gmem]" ;struct kvm_gmem *gmem; -struct inode *inode;struct file *file;int fd, err;@@ -415,32 +510,16 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) goto err_fd; } - file =anon_inode_create_getfile (anon_name, &kvm_gmem_fops, gmem, -O_RDWR ,NULL ); + file =kvm_gmem_inode_create_getfile (gmem, size, flags);if (IS_ERR (file)) { err =PTR_ERR (file); goto err_gmem; } - file->f_flags |=O_LARGEFILE ; - - inode = file->f_inode; -WARN_ON (file->f_mapping != inode->i_mapping); - - inode->i_private = (void *)(unsigned long )flags; - inode->i_op = &kvm_gmem_iops; - inode->i_mapping->a_ops = &kvm_gmem_aops; - inode->i_mode |=S_IFREG ; - inode->i_size = size; -mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); -mapping_set_inaccessible (inode->i_mapping); -/* Unmovable mappings are supposed to be marked unevictable as well. */
virt/kvm/guest_memfd.c static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) · line 525
Quoted 10 lines
virt/kvm/guest_memfd.c +1
-3-WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); -kvm_get_kvm (kvm); gmem->kvm = kvm;xa_init (&gmem->bindings); -list_add (&gmem->entry, &inode->i_mapping->i_private_list); +list_add (&gmem->entry, &file_inode (file)->i_mapping->i_private_list);fd_install (fd, file);return fd;
Thanks,
Gavin
Re: [RFC PATCH v5 02/15] KVM: guest_memfd: Make guest mem use guest mem inodes instead of anonymous inodes
Hi Gavin,
Quoted 181 lines · Gavin Shan
Hi Fuad,
Quoted 34 lines · Fuad Tabba
From: Ackerley Tng <[email protected]>
Using guest mem inodes allows us to store metadata for the backing memory on the inode. Metadata will be added in a later patch to support HugeTLB pages.
Metadata about backing memory should not be stored on the file, since the file represents a guest_memfd's binding with a struct kvm, and metadata about backing memory is not unique to a specific binding and struct kvm.
Signed-off-by Ackerley Tng <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/uapi/linux/magic.h | 1 + virt/kvm/guest_memfd.c | 119 ++++++++++++++++++++++++++++++------- 2 files changed, 100 insertions(+), 20 deletions(-)include/uapi/linux/magic.h +1
-0@@ -103,5 +103,6 @@#define DEVMEM_MAGIC 0x454d444d/* "DMEM" */ #define SECRETMEM_MAGIC 0x5345434d/* "SECM" */ #define PID_FS_MAGIC 0x50494446/* "PIDF" */ +#define GUEST_MEMORY_MAGIC 0x474d454d/* "GMEM" */ #endif /* __LINUX_MAGIC_H__ */ virt/kvm/guest_memfd.c +0
-0virt/kvm/guest_memfd.c line 3
Quoted 3 lines
virt/kvm/guest_memfd.c +2
-0@@ -1,12 +1,17 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/fs.h> +#include <linux/mount.h> This can be dropped since "
linux/mount.h" has been included to "linux/fs.h".Quoted 96 lines
virt/kvm/guest_memfd.c +67
-0#include <linux/backing-dev.h> #include <linux/falloc.h> #include <linux/kvm_host.h> +#include <linux/pseudo_fs.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> #include "kvm_mm.h" +static struct vfsmount *kvm_gmem_mnt; +struct kvm_gmem {struct kvm *kvm;struct xarray bindings;@@ -307,6 +312,38 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static const struct super_operations kvm_gmem_super_operations = { + .statfs = simple_statfs, +}; + +static int kvm_gmem_init_fs_context (struct fs_context *fc) +{ +struct pseudo_fs_context *ctx; + +if (!init_pseudo (fc,GUEST_MEMORY_MAGIC )) +return -ENOMEM ; + + ctx = fc->fs_private; + ctx->ops = &kvm_gmem_super_operations; + +return 0 ; +} + +static struct file_system_type kvm_gmem_fs = { + .name ="kvm_guest_memory" , + .init_fs_context = kvm_gmem_init_fs_context, + .kill_sb = kill_anon_super, +}; + +static void kvm_gmem_init_mount (void ) +{ + kvm_gmem_mnt =kern_mount (&kvm_gmem_fs); +BUG_ON (IS_ERR (kvm_gmem_mnt)); + +/* For giggles. Userspace can never map this anyways. */ + kvm_gmem_mnt->mnt_flags |=MNT_NOEXEC ; +} +static struct file_operations kvm_gmem_fops = { .open = generic_file_open, .release = kvm_gmem_release,@@ -316,6 +353,8 @@ static struct file_operations kvm_gmem_fops = {void kvm_gmem_init (struct module *module) { kvm_gmem_fops.owner = module; + +kvm_gmem_init_mount (); } static int kvm_gmem_migrate_folio(struct address_space *mapping,@@ -397,11 +436,67 @@ static const struct inode_operations kvm_gmem_iops = { .setattr = kvm_gmem_setattr, }; +static struct inode *kvm_gmem_inode_make_secure_inode (const char *name, +loff_t size,u64 flags) +{ +const struct qstr qname =QSTR_INIT (name,strlen (name)); +struct inode *inode; +int err; + + inode =alloc_anon_inode (kvm_gmem_mnt->mnt_sb); +if (IS_ERR (inode)) +return inode; + + err =security_inode_init_security_anon (inode, &qname,NULL ); +if (err) { +iput (inode); +return ERR_PTR (err); + } + + inode->i_private = (void *)(unsigned long )flags; + inode->i_op = &kvm_gmem_iops; + inode->i_mapping->a_ops = &kvm_gmem_aops; + inode->i_mode |=S_IFREG ; + inode->i_size = size; +mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); +mapping_set_inaccessible (inode->i_mapping); +/* Unmovable mappings are supposed to be marked unevictable as well. */ +WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); + +return inode; +} +virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 478
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+static struct file *kvm_gmem_inode_create_getfile (void *priv,loff_t size, +u64 flags) +{ +static const char *name ="[kvm-gmem]" ; +struct inode *inode; +struct file *file; + +if (kvm_gmem_fops.owner && !try_module_get (kvm_gmem_fops.owner)) +return ERR_PTR (-ENOENT ); +The validation on 'kvm_gmem_fops.owner' can be removed since
try_module_get()andmodule_put()are friendly to a NULL parameter, even whenCONFIG_MODULE_UNLOAD== NA
module_put(kvm_gmem_fops.owner)is needed in the various erroneous cases in this function. Otherwise, the reference count of the owner (module) will become imbalanced on any errors.virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 482
Quoted 4 lines
virt/kvm/guest_memfd.c +4
-0+ inode =kvm_gmem_inode_make_secure_inode (name, size, flags); +if (IS_ERR (inode)) +return ERR_CAST (inode); +
ERR_CASTmay be dropped since there is nothing to be casted or converted?virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 493
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ file =alloc_file_pseudo (inode, kvm_gmem_mnt, name,O_RDWR , + &kvm_gmem_fops); +if (IS_ERR (file)) { +iput (inode); +return file; + } + + file->f_mapping = inode->i_mapping; + file->f_flags |=O_LARGEFILE ; + file->private_data = priv; +'
file->f_mapping=inode->i_mapping' may be dropped since it's already correctly set byalloc_file_pseudo().
Quoted 5 lines
alloc_file_pseudo
alloc_path_pseudo
alloc_file
alloc_empty_file
file_init_path// Set by this function
Thanks for the fixes. Will include them when we respin.
Cheers,
/fuad
Quoted 50 lines
Quoted 36 lines
virt/kvm/guest_memfd.c +4
-17+return file; +} +static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) { -const char *anon_name ="[kvm-gmem]" ;struct kvm_gmem *gmem; -struct inode *inode;struct file *file;int fd, err;@@ -415,32 +510,16 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) goto err_fd; } - file =anon_inode_create_getfile (anon_name, &kvm_gmem_fops, gmem, -O_RDWR ,NULL ); + file =kvm_gmem_inode_create_getfile (gmem, size, flags);if (IS_ERR (file)) { err =PTR_ERR (file); goto err_gmem; } - file->f_flags |=O_LARGEFILE ; - - inode = file->f_inode; -WARN_ON (file->f_mapping != inode->i_mapping); - - inode->i_private = (void *)(unsigned long )flags; - inode->i_op = &kvm_gmem_iops; - inode->i_mapping->a_ops = &kvm_gmem_aops; - inode->i_mode |=S_IFREG ; - inode->i_size = size; -mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); -mapping_set_inaccessible (inode->i_mapping); -/* Unmovable mappings are supposed to be marked unevictable as well. */ virt/kvm/guest_memfd.c static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) · line 525
Quoted 10 lines
virt/kvm/guest_memfd.c +1
-3-WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); -kvm_get_kvm (kvm); gmem->kvm = kvm;xa_init (&gmem->bindings); -list_add (&gmem->entry, &inode->i_mapping->i_private_list); +list_add (&gmem->entry, &file_inode (file)->i_mapping->i_private_list);fd_install (fd, file);return fd;Thanks,
Gavin
Re: [RFC PATCH v5 02/15] KVM: guest_memfd: Make guest mem use guest mem inodes instead of anonymous inodes
In reply to Gavin ShanThanks for reviewing, Gavin! I'll also adopt these when I respin.
Quoted 153 lines · Gavin Shan
Hi Fuad,
Quoted 34 lines · Fuad Tabba
From: Ackerley Tng <[email protected]>
Using guest mem inodes allows us to store metadata for the backing memory on the inode. Metadata will be added in a later patch to support HugeTLB pages.
Metadata about backing memory should not be stored on the file, since the file represents a guest_memfd's binding with a struct kvm, and metadata about backing memory is not unique to a specific binding and struct kvm.
Signed-off-by Ackerley Tng <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- include/uapi/linux/magic.h | 1 + virt/kvm/guest_memfd.c | 119 ++++++++++++++++++++++++++++++------- 2 files changed, 100 insertions(+), 20 deletions(-)include/uapi/linux/magic.h +1
-0@@ -103,5 +103,6 @@#define DEVMEM_MAGIC 0x454d444d/* "DMEM" */ #define SECRETMEM_MAGIC 0x5345434d/* "SECM" */ #define PID_FS_MAGIC 0x50494446/* "PIDF" */ +#define GUEST_MEMORY_MAGIC 0x474d454d/* "GMEM" */ #endif /* __LINUX_MAGIC_H__ */ virt/kvm/guest_memfd.c +0
-0virt/kvm/guest_memfd.c line 3
Quoted 3 lines
virt/kvm/guest_memfd.c +2
-0@@ -1,12 +1,17 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/fs.h> +#include <linux/mount.h> This can be dropped since "
linux/mount.h" has been included to "linux/fs.h".Quoted 96 lines
virt/kvm/guest_memfd.c +67
-0#include <linux/backing-dev.h> #include <linux/falloc.h> #include <linux/kvm_host.h> +#include <linux/pseudo_fs.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> #include "kvm_mm.h" +static struct vfsmount *kvm_gmem_mnt; +struct kvm_gmem {struct kvm *kvm;struct xarray bindings;@@ -307,6 +312,38 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static const struct super_operations kvm_gmem_super_operations = { + .statfs = simple_statfs, +}; + +static int kvm_gmem_init_fs_context (struct fs_context *fc) +{ +struct pseudo_fs_context *ctx; + +if (!init_pseudo (fc,GUEST_MEMORY_MAGIC )) +return -ENOMEM ; + + ctx = fc->fs_private; + ctx->ops = &kvm_gmem_super_operations; + +return 0 ; +} + +static struct file_system_type kvm_gmem_fs = { + .name ="kvm_guest_memory" , + .init_fs_context = kvm_gmem_init_fs_context, + .kill_sb = kill_anon_super, +}; + +static void kvm_gmem_init_mount (void ) +{ + kvm_gmem_mnt =kern_mount (&kvm_gmem_fs); +BUG_ON (IS_ERR (kvm_gmem_mnt)); + +/* For giggles. Userspace can never map this anyways. */ + kvm_gmem_mnt->mnt_flags |=MNT_NOEXEC ; +} +static struct file_operations kvm_gmem_fops = { .open = generic_file_open, .release = kvm_gmem_release,@@ -316,6 +353,8 @@ static struct file_operations kvm_gmem_fops = {void kvm_gmem_init (struct module *module) { kvm_gmem_fops.owner = module; + +kvm_gmem_init_mount (); } static int kvm_gmem_migrate_folio(struct address_space *mapping,@@ -397,11 +436,67 @@ static const struct inode_operations kvm_gmem_iops = { .setattr = kvm_gmem_setattr, }; +static struct inode *kvm_gmem_inode_make_secure_inode (const char *name, +loff_t size,u64 flags) +{ +const struct qstr qname =QSTR_INIT (name,strlen (name)); +struct inode *inode; +int err; + + inode =alloc_anon_inode (kvm_gmem_mnt->mnt_sb); +if (IS_ERR (inode)) +return inode; + + err =security_inode_init_security_anon (inode, &qname,NULL ); +if (err) { +iput (inode); +return ERR_PTR (err); + } + + inode->i_private = (void *)(unsigned long )flags; + inode->i_op = &kvm_gmem_iops; + inode->i_mapping->a_ops = &kvm_gmem_aops; + inode->i_mode |=S_IFREG ; + inode->i_size = size; +mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); +mapping_set_inaccessible (inode->i_mapping); +/* Unmovable mappings are supposed to be marked unevictable as well. */ +WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); + +return inode; +} +virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 478
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+static struct file *kvm_gmem_inode_create_getfile (void *priv,loff_t size, +u64 flags) +{ +static const char *name ="[kvm-gmem]" ; +struct inode *inode; +struct file *file; + +if (kvm_gmem_fops.owner && !try_module_get (kvm_gmem_fops.owner)) +return ERR_PTR (-ENOENT ); +The validation on 'kvm_gmem_fops.owner' can be removed since
try_module_get()andmodule_put()are friendly to a NULL parameter, even whenCONFIG_MODULE_UNLOAD== N
Quoted 4 lines
A
module_put(kvm_gmem_fops.owner)is needed in the various erroneous cases in this function. Otherwise, the reference count of the owner (module) will become imbalanced on any errors.
Thanks for catching this! Will add module_put() for error paths.
Quoted 8 lines
virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 482
Quoted 5 lines
virt/kvm/guest_memfd.c +4
-0+ inode =kvm_gmem_inode_make_secure_inode (name, size, flags); +if (IS_ERR (inode)) +return ERR_CAST (inode); +
ERR_CASTmay be dropped since there is nothing to be casted or converted?
This cast is necessary as it casts from a struct inode * to a struct file *.
Quoted 15 lines
virt/kvm/guest_memfd.c static const struct inode_operations kvm_gmem_iops = { · line 493
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ file =alloc_file_pseudo (inode, kvm_gmem_mnt, name,O_RDWR , + &kvm_gmem_fops); +if (IS_ERR (file)) { +iput (inode); +return file; + } + + file->f_mapping = inode->i_mapping; + file->f_flags |=O_LARGEFILE ; + file->private_data = priv; +'
file->f_mapping=inode->i_mapping' may be dropped since it's already correctly set byalloc_file_pseudo().
Quoted 6 lines
alloc_file_pseudo
alloc_path_pseudo
alloc_file
alloc_empty_file
file_init_path// Set by this function
Thanks!
Quoted 50 lines
Quoted 37 lines
virt/kvm/guest_memfd.c +4
-17+return file; +} +static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) { -const char *anon_name ="[kvm-gmem]" ;struct kvm_gmem *gmem; -struct inode *inode;struct file *file;int fd, err;@@ -415,32 +510,16 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) goto err_fd; } - file =anon_inode_create_getfile (anon_name, &kvm_gmem_fops, gmem, -O_RDWR ,NULL ); + file =kvm_gmem_inode_create_getfile (gmem, size, flags);if (IS_ERR (file)) { err =PTR_ERR (file); goto err_gmem; } - file->f_flags |=O_LARGEFILE ; - - inode = file->f_inode; -WARN_ON (file->f_mapping != inode->i_mapping); - - inode->i_private = (void *)(unsigned long )flags; - inode->i_op = &kvm_gmem_iops; - inode->i_mapping->a_ops = &kvm_gmem_aops; - inode->i_mode |=S_IFREG ; - inode->i_size = size; -mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); -mapping_set_inaccessible (inode->i_mapping); -/* Unmovable mappings are supposed to be marked unevictable as well. */ virt/kvm/guest_memfd.c static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) · line 525
Quoted 10 lines
virt/kvm/guest_memfd.c +1
-3-WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping)); -kvm_get_kvm (kvm); gmem->kvm = kvm;xa_init (&gmem->bindings); -list_add (&gmem->entry, &inode->i_mapping->i_private_list); +list_add (&gmem->entry, &file_inode (file)->i_mapping->i_private_list);fd_install (fd, file);return fd;Thanks,
Gavin
RFC PATCH v5 3/15 KVM: guest_memfd: Introduce kvm_gmem_get_pfn_locked(), which retains the folio lock
In reply to Fuad TabbaCreate a new variant of kvm_gmem_get_pfn(), which retains the folio lock if it returns successfully. This is needed in subsequent patches in order to protect against races when checking whether a folio can be mapped by the host.
--- include/linux/kvm_host.h | 11 +++++++++++ virt/kvm/guest_memfd.c | 27 ++++++++++++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +11 -0
@@ -2500,6 +2500,9 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,kvm_pfn_t *pfn,struct page **page,int *max_order); +int kvm_gmem_get_pfn_locked (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,struct page **page, +int *max_order);#else static inline int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn,
@@ -2509,6 +2512,14 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,KVM_BUG_ON (1 , kvm); return -EIO; } +static inline int kvm_gmem_get_pfn_locked (struct kvm *kvm, +struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn, +struct page **page,int *max_order) +{ +KVM_BUG_ON (1 , kvm); +return -EIO ; +}#endif /* CONFIG_KVM_PRIVATE_MEM */ #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
virt/kvm/guest_memfd.c +20 -7
@@ -672,9 +672,9 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,return folio; } -int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot, -gfn_t gfn,kvm_pfn_t *pfn,struct page **page, -int *max_order) +int kvm_gmem_get_pfn_locked (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,struct page **page, +int *max_order) {pgoff_t index =kvm_gmem_get_index (slot, gfn);struct file *file =kvm_gmem_get_file (slot);
@@ -694,17 +694,30 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,if (!is_prepared) r =kvm_gmem_prepare_folio (kvm, slot, gfn, folio); -folio_unlock (folio); - -if (!r) +if (!r) { *page =folio_file_page (folio, index); -else + }else { +folio_unlock (folio);folio_put (folio); + } out:fput (file);return r; } +EXPORT_SYMBOL_GPL (kvm_gmem_get_pfn_locked); + +int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,struct page **page, +int *max_order) +{ +int r =kvm_gmem_get_pfn_locked (kvm, slot, gfn, pfn, page, max_order); + +if (!r) +unlock_page (*page); + +return r; +}EXPORT_SYMBOL_GPL (kvm_gmem_get_pfn);#ifdef CONFIG_KVM_GENERIC_PRIVATE_MEM
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 4/15 KVM: guest_memfd: Track mappability within a struct kvm_gmem_private
In reply to Fuad TabbaFrom: Ackerley Tng <[email protected]>
Track whether guest_memfd memory can be mapped within the inode, since it is property of the guest_memfd's memory contents.
The guest_memfd PRIVATE memory attribute is not used for two reasons. First because it reflects the userspace expectation for that memory location, and therefore can be toggled by userspace. The second is, although each guest_memfd file has a 1:1 binding with a KVM instance, the plan is to allow multiple files per inode, e.g. to allow intra-host migration to a new KVM instance, without destroying guest_memfd.
--- virt/kvm/guest_memfd.c | 56 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 5 deletions(-)
virt/kvm/guest_memfd.c +51 -5
@@ -18,6 +18,17 @@ struct kvm_gmem {struct list_head entry; }; +struct kvm_gmem_inode_private { +#ifdef CONFIG_KVM_GMEM_MAPPABLE +struct xarray mappable_offsets; +#endif +}; + +static struct kvm_gmem_inode_private *kvm_gmem_private (struct inode *inode) +{ +return inode->i_mapping->i_private_data; +} +/** * folio_file_pfn - like folio_file_page, but return a pfn. * @folio: The folio which contains this index.
@@ -312,8 +323,28 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static void kvm_gmem_evict_inode (struct inode *inode) +{ +struct kvm_gmem_inode_private *private =kvm_gmem_private (inode); + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +/* +* .evict_inode can be called before private data is set up if there are +* issues during inode creation. +*/ +if (private) +xa_destroy (&private->mappable_offsets); +#endif + +truncate_inode_pages_final (inode->i_mapping); + +kfree (private); +clear_inode (inode); +} +static const struct super_operations kvm_gmem_super_operations = { - .statfs = simple_statfs, + .statfs = simple_statfs, + .evict_inode = kvm_gmem_evict_inode, };static int kvm_gmem_init_fs_context (struct fs_context *fc)
@@ -440,6 +471,7 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, loff_t size, u64 flags) {const struct qstr qname =QSTR_INIT (name,strlen (name)); +struct kvm_gmem_inode_private *private;struct inode *inode;int err;
@@ -448,10 +480,19 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,return inode; err =security_inode_init_security_anon (inode, &qname,NULL ); -if (err) { -iput (inode); -return ERR_PTR (err); - } +if (err) + goto out; + + err = -ENOMEM ; + private =kzalloc (sizeof (*private),GFP_KERNEL ); +if (!private) + goto out; + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +xa_init (&private->mappable_offsets); +#endif + + inode->i_mapping->i_private_data = private; inode->i_private = (void *)(unsigned long )flags; inode->i_op = &kvm_gmem_iops;
@@ -464,6 +505,11 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping));return inode; + +out: +iput (inode); + +return ERR_PTR (err); } static struct file *kvm_gmem_inode_create_getfile(void *priv, loff_t size,
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
Re: [RFC PATCH v5 04/15] KVM: guest_memfd: Track mappability within a struct kvm_gmem_private
Hi Fuad,
Quoted 63 lines · Fuad Tabba
From: Ackerley Tng <[email protected]>
Track whether
guest_memfdmemory can be mapped within the inode, since it is property of the guest_memfd's memory contents.The
guest_memfdPRIVATE memory attribute is not used for two reasons. First because it reflects the userspace expectation for that memory location, and therefore can be toggled by userspace. The second is, although eachguest_memfdfile has a 1:1 binding with a KVM instance, the plan is to allow multiple files per inode, e.g. to allow intra-host migration to a new KVM instance, without destroyingguest_memfd.Signed-off-by Ackerley Tng <[email protected]>Co-developed-by Vishal Annapurve <[email protected]>Signed-off-by Vishal Annapurve <[email protected]>Co-developed-by Fuad Tabba <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- virt/kvm/guest_memfd.c | 56 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 5 deletions(-)virt/kvm/guest_memfd.c +26
-0@@ -18,6 +18,17 @@ struct kvm_gmem {struct list_head entry; }; +struct kvm_gmem_inode_private { +#ifdef CONFIG_KVM_GMEM_MAPPABLE +struct xarray mappable_offsets; +#endif +}; + +static struct kvm_gmem_inode_private *kvm_gmem_private (struct inode *inode) +{ +return inode->i_mapping->i_private_data; +} +/** * folio_file_pfn - like folio_file_page, but return a pfn. * @folio: The folio which contains this index. @@ -312,8 +323,28 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static void kvm_gmem_evict_inode (struct inode *inode) +{ +struct kvm_gmem_inode_private *private =kvm_gmem_private (inode); + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +/* +* .evict_inode can be called before private data is set up if there are +* issues during inode creation. +*/ +if (private) +xa_destroy (&private->mappable_offsets); +#endif + +truncate_inode_pages_final (inode->i_mapping); +
virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 349
Quoted 10 lines
virt/kvm/guest_memfd.c +6
-1+kfree (private); +clear_inode (inode); +} +static const struct super_operations kvm_gmem_super_operations = { - .statfs = simple_statfs, + .statfs = simple_statfs, + .evict_inode = kvm_gmem_evict_inode, };
As I understood, ->destroy_inode() may be more suitable place where the xarray is released. ->evict_inode() usually detach the inode from the existing struct, to make it offline. ->destroy_inode() is actually the place where the associated resource (memory) is relased.
Another benefit with ->destroy_inode() is we're not concerned to truncate_inode_pages_final() and clear_inode().
Quoted 21 lines
virt/kvm/guest_memfd.c +5
-4static int kvm_gmem_init_fs_context (struct fs_context *fc)@@ -440,6 +471,7 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, loff_t size, u64 flags) {const struct qstr qname =QSTR_INIT (name,strlen (name)); +struct kvm_gmem_inode_private *private;struct inode *inode;int err;@@ -448,10 +480,19 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,return inode; err =security_inode_init_security_anon (inode, &qname,NULL ); -if (err) { -iput (inode); -return ERR_PTR (err); - } +if (err) + goto out; + + err = -ENOMEM ;
virt/kvm/guest_memfd.c static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, · line 496
Quoted 10 lines
virt/kvm/guest_memfd.c +9
-0+ private =kzalloc (sizeof (*private),GFP_KERNEL ); +if (!private) + goto out; + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +xa_init (&private->mappable_offsets); +#endif + + inode->i_mapping->i_private_data = private;
The whole block of code needs to be guarded by CONFIG_KVM_GMEM_MAPPABLE because kzalloc(sizeof(...)) is translated to kzalloc(0) when CONFIG_KVM_GMEM_MAPPABLE is disabled, and kzalloc() will always fail. It will lead to unusable guest-memfd if CONFIG_KVM_GMEM_MAPPABLE is disabled.
Quoted 4 lines
virt/kvm/guest_memfd.c +0
-0inode->i_private = (void *)(unsigned long )flags; inode->i_op = &kvm_gmem_iops;@@ -464,6 +505,11 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping));
virt/kvm/guest_memfd.c static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, · line 515
Quoted 10 lines
virt/kvm/guest_memfd.c +5
-0return inode; + +out: +iput (inode); + +return ERR_PTR (err); } static struct file *kvm_gmem_inode_create_getfile(void *priv, loff_t size,
Thanks,
Gavin
Re: [RFC PATCH v5 04/15] KVM: guest_memfd: Track mappability within a struct kvm_gmem_private
Hi Gavin,
Quoted 83 lines · Gavin Shan
Hi Fuad,
Quoted 63 lines · Fuad Tabba
From: Ackerley Tng <[email protected]>
Track whether
guest_memfdmemory can be mapped within the inode, since it is property of the guest_memfd's memory contents.The
guest_memfdPRIVATE memory attribute is not used for two reasons. First because it reflects the userspace expectation for that memory location, and therefore can be toggled by userspace. The second is, although eachguest_memfdfile has a 1:1 binding with a KVM instance, the plan is to allow multiple files per inode, e.g. to allow intra-host migration to a new KVM instance, without destroyingguest_memfd.Signed-off-by Ackerley Tng <[email protected]>Co-developed-by Vishal Annapurve <[email protected]>Signed-off-by Vishal Annapurve <[email protected]>Co-developed-by Fuad Tabba <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- virt/kvm/guest_memfd.c | 56 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 5 deletions(-)virt/kvm/guest_memfd.c +26
-0@@ -18,6 +18,17 @@ struct kvm_gmem {struct list_head entry; }; +struct kvm_gmem_inode_private { +#ifdef CONFIG_KVM_GMEM_MAPPABLE +struct xarray mappable_offsets; +#endif +}; + +static struct kvm_gmem_inode_private *kvm_gmem_private (struct inode *inode) +{ +return inode->i_mapping->i_private_data; +} +/** * folio_file_pfn - like folio_file_page, but return a pfn. * @folio: The folio which contains this index. @@ -312,8 +323,28 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static void kvm_gmem_evict_inode (struct inode *inode) +{ +struct kvm_gmem_inode_private *private =kvm_gmem_private (inode); + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +/* +* .evict_inode can be called before private data is set up if there are +* issues during inode creation. +*/ +if (private) +xa_destroy (&private->mappable_offsets); +#endif + +truncate_inode_pages_final (inode->i_mapping); +virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 349
Quoted 10 lines
virt/kvm/guest_memfd.c +6
-1+kfree (private); +clear_inode (inode); +} +static const struct super_operations kvm_gmem_super_operations = { - .statfs = simple_statfs, + .statfs = simple_statfs, + .evict_inode = kvm_gmem_evict_inode, };As I understood, ->destroy_inode() may be more suitable place where the xarray is released. ->evict_inode() usually detach the inode from the existing struct, to make it offline. ->destroy_inode() is actually the place where the associated resource (memory) is relased.
Quoted 2 lines
Another benefit with ->destroy_inode() is we're not concerned to
truncate_inode_pages_final()andclear_inode().
I see. I'll give this a try.
Quoted 29 lines
Quoted 29 lines
virt/kvm/guest_memfd.c +12
-4static int kvm_gmem_init_fs_context (struct fs_context *fc)@@ -440,6 +471,7 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, loff_t size, u64 flags) {const struct qstr qname =QSTR_INIT (name,strlen (name)); +struct kvm_gmem_inode_private *private;struct inode *inode;int err;@@ -448,10 +480,19 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,return inode; err =security_inode_init_security_anon (inode, &qname,NULL ); -if (err) { -iput (inode); -return ERR_PTR (err); - } +if (err) + goto out; + + err = -ENOMEM ; + private =kzalloc (sizeof (*private),GFP_KERNEL ); +if (!private) + goto out; + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +xa_init (&private->mappable_offsets); +#endif
Quoted 8 lines
virt/kvm/guest_memfd.c static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, · line 496
Quoted 3 lines
virt/kvm/guest_memfd.c +2
-0+ + inode->i_mapping->i_private_data = private;The whole block of code needs to be guarded by
CONFIG_KVM_GMEM_MAPPABLEbecause kzalloc(sizeof(...)) is translated to kzalloc(0) whenCONFIG_KVM_GMEM_MAPPABLEis disabled, andkzalloc()will always fail. It will lead to unusable guest-memfd ifCONFIG_KVM_GMEM_MAPPABLEis disabled.
Good point, thanks for pointing this out.
Cheers,
/fuad
Quoted 18 lines
Quoted 4 lines
virt/kvm/guest_memfd.c +0
-0inode->i_private = (void *)(unsigned long )flags; inode->i_op = &kvm_gmem_iops;@@ -464,6 +505,11 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping));virt/kvm/guest_memfd.c static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, · line 515
Quoted 10 lines
virt/kvm/guest_memfd.c +5
-0return inode; + +out: +iput (inode); + +return ERR_PTR (err); } static struct file *kvm_gmem_inode_create_getfile(void *priv, loff_t size,Thanks,
Gavin
Re: [RFC PATCH v5 04/15] KVM: guest_memfd: Track mappability within a struct kvm_gmem_private
Quoted 86 lines · Fuad Tabba
Hi Gavin,
Quoted 83 lines · Gavin Shan
Hi Fuad,
Quoted 63 lines · Fuad Tabba
From: Ackerley Tng <[email protected]>
Track whether
guest_memfdmemory can be mapped within the inode, since it is property of the guest_memfd's memory contents.The
guest_memfdPRIVATE memory attribute is not used for two reasons. First because it reflects the userspace expectation for that memory location, and therefore can be toggled by userspace. The second is, although eachguest_memfdfile has a 1:1 binding with a KVM instance, the plan is to allow multiple files per inode, e.g. to allow intra-host migration to a new KVM instance, without destroyingguest_memfd.Signed-off-by Ackerley Tng <[email protected]>Co-developed-by Vishal Annapurve <[email protected]>Signed-off-by Vishal Annapurve <[email protected]>Co-developed-by Fuad Tabba <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- virt/kvm/guest_memfd.c | 56 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 5 deletions(-)virt/kvm/guest_memfd.c +26
-0@@ -18,6 +18,17 @@ struct kvm_gmem {struct list_head entry; }; +struct kvm_gmem_inode_private { +#ifdef CONFIG_KVM_GMEM_MAPPABLE +struct xarray mappable_offsets; +#endif +}; + +static struct kvm_gmem_inode_private *kvm_gmem_private (struct inode *inode) +{ +return inode->i_mapping->i_private_data; +} +/** * folio_file_pfn - like folio_file_page, but return a pfn. * @folio: The folio which contains this index. @@ -312,8 +323,28 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)return gfn - slot->base_gfn + slot->gmem.pgoff; } +static void kvm_gmem_evict_inode (struct inode *inode) +{ +struct kvm_gmem_inode_private *private =kvm_gmem_private (inode); + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +/* +* .evict_inode can be called before private data is set up if there are +* issues during inode creation. +*/ +if (private) +xa_destroy (&private->mappable_offsets); +#endif + +truncate_inode_pages_final (inode->i_mapping); +virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 349
Quoted 10 lines
virt/kvm/guest_memfd.c +6
-1+kfree (private); +clear_inode (inode); +} +static const struct super_operations kvm_gmem_super_operations = { - .statfs = simple_statfs, + .statfs = simple_statfs, + .evict_inode = kvm_gmem_evict_inode, };As I understood, ->destroy_inode() may be more suitable place where the xarray is released. ->evict_inode() usually detach the inode from the existing struct, to make it offline. ->destroy_inode() is actually the place where the associated resource (memory) is relased.
Quoted 5 lines
Quoted 2 lines
Another benefit with ->destroy_inode() is we're not concerned to
truncate_inode_pages_final()andclear_inode().I see. I'll give this a try.
While working on 1G page support (old revision at [1]), I was looking at this.
Using .destroy_inode to clean up private->mappable_offsets should work fine, and I agree this should be refactored to use .destroy_inode instead. Thanks for pointing this out!
FWIW, for 1G page support, the truncation process has to be overridden to, so the .evict_inode override will have to come back.
Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
virt/kvm/guest_memfd.c +12
-4static int kvm_gmem_init_fs_context (struct fs_context *fc)@@ -440,6 +471,7 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, loff_t size, u64 flags) {const struct qstr qname =QSTR_INIT (name,strlen (name)); +struct kvm_gmem_inode_private *private;struct inode *inode;int err;@@ -448,10 +480,19 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,return inode; err =security_inode_init_security_anon (inode, &qname,NULL ); -if (err) { -iput (inode); -return ERR_PTR (err); - } +if (err) + goto out; + + err = -ENOMEM ; + private =kzalloc (sizeof (*private),GFP_KERNEL ); +if (!private) + goto out; + +#ifdef CONFIG_KVM_GMEM_MAPPABLE +xa_init (&private->mappable_offsets); +#endif Quoted 8 lines
virt/kvm/guest_memfd.c static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, · line 496
Quoted 3 lines
virt/kvm/guest_memfd.c +2
-0+ + inode->i_mapping->i_private_data = private;The whole block of code needs to be guarded by
CONFIG_KVM_GMEM_MAPPABLEbecause kzalloc(sizeof(...)) is translated to kzalloc(0) whenCONFIG_KVM_GMEM_MAPPABLEis disabled, andkzalloc()will always fail. It will lead to unusable guest-memfd ifCONFIG_KVM_GMEM_MAPPABLEis disabled.
Quoted 23 lines
Good point, thanks for pointing this out.
Cheers,
/fuadQuoted 18 lines
Quoted 4 lines
virt/kvm/guest_memfd.c +0
-0inode->i_private = (void *)(unsigned long )flags; inode->i_op = &kvm_gmem_iops;@@ -464,6 +505,11 @@ static struct inode *kvm_gmem_inode_make_secure_inode(const char *name,WARN_ON_ONCE (!mapping_unevictable (inode->i_mapping));virt/kvm/guest_memfd.c static struct inode *kvm_gmem_inode_make_secure_inode(const char *name, · line 515
Quoted 10 lines
virt/kvm/guest_memfd.c +5
-0return inode; + +out: +iput (inode); + +return ERR_PTR (err); } static struct file *kvm_gmem_inode_create_getfile(void *priv, loff_t size,Thanks,
Gavin
RFC PATCH v5 5/15 KVM: guest_memfd: Folio mappability states and functions that manage their transition
In reply to Fuad TabbaTo allow restricted mapping of guest_memfd folios by the host, guest_memfd needs to track whether they can be mapped and by who, since the mapping will only be allowed under conditions where it safe to access these folios. These conditions depend on the folios being explicitly shared with the host, or not yet exposed to the guest (e.g., at initialization).
This patch introduces states that determine whether the host and the guest can fault in the folios as well as the functions that manage transitioning between those states.
--- include/linux/kvm_host.h | 53 ++++++++++++++ virt/kvm/guest_memfd.c | 153 +++++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 92 +++++++++++++++++++++++ 3 files changed, 298 insertions(+)
include/linux/kvm_host.h +53 -0
@@ -2564,4 +2564,57 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range);#endif +#ifdef CONFIG_KVM_GMEM_MAPPABLE +bool kvm_gmem_is_mappable (struct kvm *kvm,gfn_t gfn,gfn_t end); +int kvm_gmem_set_mappable (struct kvm *kvm,gfn_t start,gfn_t end); +int kvm_gmem_clear_mappable (struct kvm *kvm,gfn_t start,gfn_t end); +int kvm_slot_gmem_set_mappable (struct kvm_memory_slot *slot,gfn_t start, +gfn_t end); +int kvm_slot_gmem_clear_mappable (struct kvm_memory_slot *slot,gfn_t start, +gfn_t end); +bool kvm_slot_gmem_is_mappable (struct kvm_memory_slot *slot,gfn_t gfn); +bool kvm_slot_gmem_is_guest_mappable (struct kvm_memory_slot *slot,gfn_t gfn); +#else +static inline bool kvm_gmem_is_mappable (struct kvm *kvm,gfn_t gfn,gfn_t end) +{ +WARN_ON_ONCE (1 ); +return false; +} +static inline int kvm_gmem_set_mappable (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (1 ); +return -EINVAL ; +} +static inline int kvm_gmem_clear_mappable (struct kvm *kvm,gfn_t start, +gfn_t end) +{ +WARN_ON_ONCE (1 ); +return -EINVAL ; +} +static inline int kvm_slot_gmem_set_mappable (struct kvm_memory_slot *slot, +gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (1 ); +return -EINVAL ; +} +static inline int kvm_slot_gmem_clear_mappable (struct kvm_memory_slot *slot, +gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (1 ); +return -EINVAL ; +} +static inline bool kvm_slot_gmem_is_mappable (struct kvm_memory_slot *slot, +gfn_t gfn) +{ +WARN_ON_ONCE (1 ); +return false; +} +static inline bool kvm_slot_gmem_is_guest_mappable (struct kvm_memory_slot *slot, +gfn_t gfn) +{ +WARN_ON_ONCE (1 ); +return false; +} +#endif /* CONFIG_KVM_GMEM_MAPPABLE */ +#endif
virt/kvm/guest_memfd.c +153 -0
159 lines not on this page · raw message
virt/kvm/kvm_main.c +92 -0
98 lines not on this page · raw message
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
Re: [RFC PATCH v5 05/15] KVM: guest_memfd: Folio mappability states and functions that manage their transition
Quoted 28 lines · Fuad Tabba
virt/kvm/guest_memfd.c +28
-0+/* +* Marks the range [start, end) as not mappable by the host. If the host doesn't +* have any references to a particular folio, then that folio is marked as +* mappable by the guest. +* +* However, if the host still has references to the folio, then the folio is +* marked and not mappable by anyone. Marking it is not mappable allows it to +* drain all references from the host, and to ensure that the hypervisor does +* not transition the folio to private, since the host still might access it. +* +* Usually called when guest unshares memory with the host. +*/ +static int gmem_clear_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +void *xval_none =xa_mk_value (KVM_GMEM_NONE_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) { +struct folio *folio; +int refcount =0 ; + + folio =filemap_lock_folio (inode->i_mapping, i); +if (!IS_ERR (folio)) { + refcount =folio_ref_count (folio);
virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 449
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+ }else { + r =PTR_ERR (folio); +if (WARN_ON_ONCE (r != -ENOENT )) +break ; + + folio =NULL ; + } + +/* +1 references are expected because of filemap_lock_folio(). */ +if (folio && refcount >folio_nr_pages (folio) +1 ) {
Looks racy.
What prevent anybody from obtaining a reference just after check?
Lock on folio doesn't stop random filemap_get_entry() from elevating the refcount.
folio_ref_freeze() might be required.
Quoted 25 lines
virt/kvm/guest_memfd.c +25
-0+/* +* Outstanding references, the folio cannot be faulted +* in by anyone until they're dropped. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_none,GFP_KERNEL )); + }else { +/* +* No outstanding references. Transition the folio to +* guest mappable immediately. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_guest,GFP_KERNEL )); + } + +if (folio) { +folio_unlock (folio); +folio_put (folio); + } + +if (WARN_ON_ONCE (r)) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +}
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [RFC PATCH v5 05/15] KVM: guest_memfd: Folio mappability states and functions that manage their transition
Quoted 43 lines · Kirill A. Shutemov
Quoted 28 lines · Fuad Tabba
virt/kvm/guest_memfd.c +28
-0+/* +* Marks the range [start, end) as not mappable by the host. If the host doesn't +* have any references to a particular folio, then that folio is marked as +* mappable by the guest. +* +* However, if the host still has references to the folio, then the folio is +* marked and not mappable by anyone. Marking it is not mappable allows it to +* drain all references from the host, and to ensure that the hypervisor does +* not transition the folio to private, since the host still might access it. +* +* Usually called when guest unshares memory with the host. +*/ +static int gmem_clear_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +void *xval_none =xa_mk_value (KVM_GMEM_NONE_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) { +struct folio *folio; +int refcount =0 ; + + folio =filemap_lock_folio (inode->i_mapping, i); +if (!IS_ERR (folio)) { + refcount =folio_ref_count (folio);virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 449
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+ }else { + r =PTR_ERR (folio); +if (WARN_ON_ONCE (r != -ENOENT )) +break ; + + folio =NULL ; + } + +/* +1 references are expected because of filemap_lock_folio(). */ +if (folio && refcount >folio_nr_pages (folio) +1 ) {Looks racy.
Quoted 6 lines
What prevent anybody from obtaining a reference just after check?
Lock on folio doesn't stop random
filemap_get_entry()from elevating the refcount.
folio_ref_freeze()might be required.
I thought the folio lock would be sufficient, but you're right, nothing prevents getting a reference after the check. I'll use a folio_ref_freeze() when I respin.
Thanks,
/fuad
Quoted 28 lines
Quoted 15 lines
virt/kvm/guest_memfd.c +15
-0+/* +* Outstanding references, the folio cannot be faulted +* in by anyone until they're dropped. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_none,GFP_KERNEL )); + }else { +/* +* No outstanding references. Transition the folio to +* guest mappable immediately. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_guest,GFP_KERNEL )); + } + +if (folio) { +folio_unlock (folio);virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 474
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+folio_put (folio); + } + +if (WARN_ON_ONCE (r)) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +}--
Kiryl Shutsemau / Kirill A. Shutemov
Re: [RFC PATCH v5 05/15] KVM: guest_memfd: Folio mappability states and functions that manage their transition
Quoted 49 lines · Fuad Tabba
Quoted 48 lines · Kirill A. Shutemov
Quoted 28 lines · Fuad Tabba
virt/kvm/guest_memfd.c +28
-0+/* +* Marks the range [start, end) as not mappable by the host. If the host doesn't +* have any references to a particular folio, then that folio is marked as +* mappable by the guest. +* +* However, if the host still has references to the folio, then the folio is +* marked and not mappable by anyone. Marking it is not mappable allows it to +* drain all references from the host, and to ensure that the hypervisor does +* not transition the folio to private, since the host still might access it. +* +* Usually called when guest unshares memory with the host. +*/ +static int gmem_clear_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +void *xval_none =xa_mk_value (KVM_GMEM_NONE_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) { +struct folio *folio; +int refcount =0 ; + + folio =filemap_lock_folio (inode->i_mapping, i); +if (!IS_ERR (folio)) { + refcount =folio_ref_count (folio);virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 449
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+ }else { + r =PTR_ERR (folio); +if (WARN_ON_ONCE (r != -ENOENT )) +break ; + + folio =NULL ; + } + +/* +1 references are expected because of filemap_lock_folio(). */ +if (folio && refcount >folio_nr_pages (folio) +1 ) {Looks racy.
What prevent anybody from obtaining a reference just after check?
Lock on folio doesn't stop random
filemap_get_entry()from elevating the refcount.
Quoted 9 lines
Quoted 1 line
folio_ref_freeze()might be required.I thought the folio lock would be sufficient, but you're right, nothing prevents getting a reference after the check. I'll use a
folio_ref_freeze()when I respin.Thanks,
/fuad
Is it correct to say that the only non-racy check for refcounts is a check for refcount == 0?
What do you think of this instead: If there exists a folio, don't check the refcount, just set mappability to NONE and register the callback (the folio should already have been unmapped, which leaves folio->page_type available for use), and then drop the filemap's refcounts. When the filemap's refcounts are dropped, in most cases (no transient refcounts) the callback will be hit and the callback can set mappability to GUEST.
If there are transient refcounts, the folio will just be waiting for the refcounts to drop to 0, and that's when the callback will be hit and the mappability can be transitioned to GUEST.
If there isn't a folio, then guest_memfd was requested to set mappability ahead of any folio allocation, and in that case transitioning to GUEST immediately is correct.
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
virt/kvm/guest_memfd.c +15
-0+/* +* Outstanding references, the folio cannot be faulted +* in by anyone until they're dropped. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_none,GFP_KERNEL )); + }else { +/* +* No outstanding references. Transition the folio to +* guest mappable immediately. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_guest,GFP_KERNEL )); + } + +if (folio) { +folio_unlock (folio);virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 474
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+folio_put (folio); + } + +if (WARN_ON_ONCE (r)) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +}--
Kiryl Shutsemau / Kirill A. Shutemov
Re: [RFC PATCH v5 05/15] KVM: guest_memfd: Folio mappability states and functions that manage their transition
Hi Ackerley,
Quoted 73 lines · Ackerley Tng
Quoted 49 lines · Fuad Tabba
Quoted 48 lines · Kirill A. Shutemov
Quoted 28 lines · Fuad Tabba
virt/kvm/guest_memfd.c +28
-0+/* +* Marks the range [start, end) as not mappable by the host. If the host doesn't +* have any references to a particular folio, then that folio is marked as +* mappable by the guest. +* +* However, if the host still has references to the folio, then the folio is +* marked and not mappable by anyone. Marking it is not mappable allows it to +* drain all references from the host, and to ensure that the hypervisor does +* not transition the folio to private, since the host still might access it. +* +* Usually called when guest unshares memory with the host. +*/ +static int gmem_clear_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +void *xval_none =xa_mk_value (KVM_GMEM_NONE_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) { +struct folio *folio; +int refcount =0 ; + + folio =filemap_lock_folio (inode->i_mapping, i); +if (!IS_ERR (folio)) { + refcount =folio_ref_count (folio);virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 449
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+ }else { + r =PTR_ERR (folio); +if (WARN_ON_ONCE (r != -ENOENT )) +break ; + + folio =NULL ; + } + +/* +1 references are expected because of filemap_lock_folio(). */ +if (folio && refcount >folio_nr_pages (folio) +1 ) {Looks racy.
What prevent anybody from obtaining a reference just after check?
Lock on folio doesn't stop random
filemap_get_entry()from elevating the refcount.Quoted 9 lines
Quoted 1 line
folio_ref_freeze()might be required.I thought the folio lock would be sufficient, but you're right, nothing prevents getting a reference after the check. I'll use a
folio_ref_freeze()when I respin.Thanks,
/fuadIs it correct to say that the only non-racy check for refcounts is a check for refcount == 0?
What do you think of this instead: If there exists a folio, don't check the refcount, just set mappability to NONE and register the callback (the folio should already have been unmapped, which leaves
folio->page_typeavailable for use), and then drop the filemap's refcounts. When the filemap's refcounts are dropped, in most cases (no transient refcounts) the callback will be hit and the callback can set mappability to GUEST.
Quoted 7 lines
If there are transient refcounts, the folio will just be waiting for the refcounts to drop to 0, and that's when the callback will be hit and the mappability can be transitioned to GUEST.
If there isn't a folio, then
guest_memfdwas requested to set mappability ahead of any folio allocation, and in that case transitioning to GUEST immediately is correct.
This seems to me to add additional complexity to the common case that isn't needed for correctness, and would make things more difficult to reason about. If we know that there aren't any mappings at the host (mapcount == 0), and we know that the refcount has at one point reached 0 after we have taken the folio lock, even if the refcount gets (transiently) elevated, we know that no one at the host is accessing the folio itself.
Keep in mind that the common case (in a well behaved system) is that neither the mapcount nor the refcount are elevated, and both for performance, and for understanding, I think that's what we should be targeting. Unless of course I'm wrong, and there's a correctness issue here.
Cheers,
/fuad
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
virt/kvm/guest_memfd.c +15
-0+/* +* Outstanding references, the folio cannot be faulted +* in by anyone until they're dropped. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_none,GFP_KERNEL )); + }else { +/* +* No outstanding references. Transition the folio to +* guest mappable immediately. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_guest,GFP_KERNEL )); + } + +if (folio) { +folio_unlock (folio);virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 474
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+folio_put (folio); + } + +if (WARN_ON_ONCE (r)) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +}--
Kiryl Shutsemau / Kirill A. Shutemov
Re: [RFC PATCH v5 05/15] KVM: guest_memfd: Folio mappability states and functions that manage their transition
In reply to Fuad TabbaFuad Tabba <[email protected]> writes:
This question should not block merging of this series since performance can be improved in a separate series:
Quoted 6 lines
<snip>
virt/kvm/guest_memfd.c +4
-0+ +/* +* Marks the range [start, end) as mappable by both the host and the guest. +* Usually called when guest shares memory with the host.
virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 402
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+*/ +static int gmem_set_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval =xa_mk_value (KVM_GMEM_ALL_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) {
Were any alternative data structures considered, or does anyone have suggestions for alternatives? Doing xa_store() in a loop here will take a long time for large ranges.
I looked into the following:
Option 1: (preferred) Maple trees
Maple tree has a nice API, though it would be better if it can combine ranges that have the same value.
I will have to dig into performance, but I'm assuming that even large ranges are stored in a few nodes so this would be faster than iterating over indices in an xarray.
void explore_maple_tree(void)
{
DEFINE_MTREE(mt);
mt_init_flags(&mt, MT_FLAGS_LOCK_EXTERN | MT_FLAGS_USE_RCU);
mtree_store_range(&mt, 0, 16, xa_mk_value(0x20), GFP_KERNEL);
mtree_store_range(&mt, 8, 24, xa_mk_value(0x32), GFP_KERNEL);
mtree_store_range(&mt, 5, 10, xa_mk_value(0x32), GFP_KERNEL);
{
void *entry;
MA_STATE(mas, &mt, 0, 0);
mas_for_each(&mas, entry, ULONG_MAX) {
pr_err("[%ld, %ld]: 0x%lx\n", mas.index, mas.last, xa_to_value(entry));
}
}
mtree_destroy(&mt);
}stdout:
[0, 4]: 0x20
[5, 10]: 0x32
[11, 24]: 0x32
Option 2: Multi-index xarray
The API is more complex than maple tree's, and IIUC multi-index xarrays are not generalizable to any range, so the range can't be 8 1G pages + 1 4K page for example. The size of the range has to be a power of 2 that is greater than 4K.
Using multi-index xarrays would mean computing order to store multi-index entries. This can be computed from the size of the range to be added, but is an additional source of errors.
Option 3: Interval tree, which is built on top of red-black trees
The API is set up at a lower level. A macro is used to define interval trees, the user has to deal with nodes in the tree directly and separately define functions to override sub-ranges in larger ranges.
Quoted 75 lines
virt/kvm/guest_memfd.c +73
-0+ r =xa_err (xa_store (mappable_offsets, i, xval,GFP_KERNEL )); +if (r) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Marks the range [start, end) as not mappable by the host. If the host doesn't +* have any references to a particular folio, then that folio is marked as +* mappable by the guest. +* +* However, if the host still has references to the folio, then the folio is +* marked and not mappable by anyone. Marking it is not mappable allows it to +* drain all references from the host, and to ensure that the hypervisor does +* not transition the folio to private, since the host still might access it. +* +* Usually called when guest unshares memory with the host. +*/ +static int gmem_clear_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +void *xval_none =xa_mk_value (KVM_GMEM_NONE_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) { +struct folio *folio; +int refcount =0 ; + + folio =filemap_lock_folio (inode->i_mapping, i); +if (!IS_ERR (folio)) { + refcount =folio_ref_count (folio); + }else { + r =PTR_ERR (folio); +if (WARN_ON_ONCE (r != -ENOENT )) +break ; + + folio =NULL ; + } + +/* +1 references are expected because of filemap_lock_folio(). */ +if (folio && refcount >folio_nr_pages (folio) +1 ) { +/* +* Outstanding references, the folio cannot be faulted +* in by anyone until they're dropped. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_none,GFP_KERNEL )); + }else { +/* +* No outstanding references. Transition the folio to +* guest mappable immediately. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_guest,GFP_KERNEL )); + } + +if (folio) { +folio_unlock (folio); +folio_put (folio); + } + +if (WARN_ON_ONCE (r)) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} +
Re: [RFC PATCH v5 05/15] KVM: guest_memfd: Folio mappability states and functions that manage their transition
Hi Ackerley,
Quoted 77 lines · Ackerley Tng
Fuad Tabba <[email protected]> writes:
This question should not block merging of this series since performance can be improved in a separate series:
Quoted 6 lines
<snip>
virt/kvm/guest_memfd.c +4
-0+ +/* +* Marks the range [start, end) as mappable by both the host and the guest. +* Usually called when guest shares memory with the host. virt/kvm/guest_memfd.c static void kvm_gmem_init_mount(void) · line 402
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+*/ +static int gmem_set_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval =xa_mk_value (KVM_GMEM_ALL_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) {Were any alternative data structures considered, or does anyone have suggestions for alternatives? Doing
xa_store()in a loop here will take a long time for large ranges.I looked into the following:
Option 1: (preferred) Maple trees
Maple tree has a nice API, though it would be better if it can combine ranges that have the same value.
I will have to dig into performance, but I'm assuming that even large ranges are stored in a few nodes so this would be faster than iterating over indices in an xarray.
void
explore_maple_tree(void){ DEFINE_MTREE(mt); mt_init_flags(&mt, MT_FLAGS_LOCK_EXTERN | MT_FLAGS_USE_RCU); mtree_store_range(&mt, 0, 16, xa_mk_value(0x20), GFP_KERNEL); mtree_store_range(&mt, 8, 24, xa_mk_value(0x32), GFP_KERNEL); mtree_store_range(&mt, 5, 10, xa_mk_value(0x32), GFP_KERNEL); { void *entry; MA_STATE(mas, &mt, 0, 0); mas_for_each(&mas, entry, ULONG_MAX) { pr_err("[%ld, %ld]: 0x%lx\n", mas.index, mas.last, xa_to_value(entry)); } } mtree_destroy(&mt); }stdout:
[0, 4]:
0x20
[5, 10]:0x32
[11, 24]:0x32Option 2: Multi-index xarray
The API is more complex than maple tree's, and IIUC multi-index xarrays are not generalizable to any range, so the range can't be 8 1G pages + 1 4K page for example. The size of the range has to be a power of 2 that is greater than 4K.
Using multi-index xarrays would mean computing order to store multi-index entries. This can be computed from the size of the range to be added, but is an additional source of errors.
Quoted 5 lines
Option 3: Interval tree, which is built on top of red-black trees
The API is set up at a lower level. A macro is used to define interval trees, the user has to deal with nodes in the tree directly and separately define functions to override sub-ranges in larger ranges.
I didn't consider any other data structures, mainly out of laziness :) What I mean by that is, xarrays is what is already used in guest_memfd for tracking other gfn related items, even though many have talked about in the future replacing it with something else.
I agree with you that it's not the ideal data structure, but also, like you said, this isn't part of the interface, and it would be easy to replace in the future. As you mention, one of the challenges is figuring out the performance impact in practice, and once things have settled down and the interface is more or less settled, some benchmarking would be useful to guide us here.
Thanks!
/fuad
Quoted 75 lines
Quoted 75 lines
virt/kvm/guest_memfd.c +73
-0+ r =xa_err (xa_store (mappable_offsets, i, xval,GFP_KERNEL )); +if (r) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Marks the range [start, end) as not mappable by the host. If the host doesn't +* have any references to a particular folio, then that folio is marked as +* mappable by the guest. +* +* However, if the host still has references to the folio, then the folio is +* marked and not mappable by anyone. Marking it is not mappable allows it to +* drain all references from the host, and to ensure that the hypervisor does +* not transition the folio to private, since the host still might access it. +* +* Usually called when guest unshares memory with the host. +*/ +static int gmem_clear_mappable (struct inode *inode,pgoff_t start,pgoff_t end) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +void *xval_none =xa_mk_value (KVM_GMEM_NONE_MAPPABLE ); +pgoff_t i; +int r =0 ; + +filemap_invalidate_lock (inode->i_mapping); +for (i = start; i < end; i++) { +struct folio *folio; +int refcount =0 ; + + folio =filemap_lock_folio (inode->i_mapping, i); +if (!IS_ERR (folio)) { + refcount =folio_ref_count (folio); + }else { + r =PTR_ERR (folio); +if (WARN_ON_ONCE (r != -ENOENT )) +break ; + + folio =NULL ; + } + +/* +1 references are expected because of filemap_lock_folio(). */ +if (folio && refcount >folio_nr_pages (folio) +1 ) { +/* +* Outstanding references, the folio cannot be faulted +* in by anyone until they're dropped. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_none,GFP_KERNEL )); + }else { +/* +* No outstanding references. Transition the folio to +* guest mappable immediately. +*/ + r =xa_err (xa_store (mappable_offsets, i, xval_guest,GFP_KERNEL )); + } + +if (folio) { +folio_unlock (folio); +folio_put (folio); + } + +if (WARN_ON_ONCE (r)) +break ; + } +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} +
RFC PATCH v5 6/15 KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Fuad TabbaBefore transitioning a guest_memfd folio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.
This patch introduces a new type for guest_memfd folios, and uses that to register a callback that informs the guest_memfd subsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.
---
The function kvm_slot_gmem_register_callback() isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.
[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
--- include/linux/kvm_host.h | 11 +++ include/linux/page-flags.h | 7 ++ mm/debug.c | 1 + mm/swap.c | 4 + virt/kvm/guest_memfd.c | 145 +++++++++++++++++++++++++++++++++++++ 5 files changed, 168 insertions(+)
include/linux/kvm_host.h +11 -0
@@ -2574,6 +2574,8 @@ int kvm_slot_gmem_clear_mappable(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);bool kvm_slot_gmem_is_mappable (struct kvm_memory_slot *slot,gfn_t gfn);bool kvm_slot_gmem_is_guest_mappable (struct kvm_memory_slot *slot,gfn_t gfn); +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn); +void kvm_gmem_handle_folio_put (struct folio *folio);#else static inline bool kvm_gmem_is_mappable (struct kvm *kvm,gfn_t gfn,gfn_t end) {
@@ -2615,6 +2617,15 @@ static inline bool kvm_slot_gmem_is_guest_mappable(struct kvm_memory_slot *slot,WARN_ON_ONCE (1 );return false; } +static inline int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +WARN_ON_ONCE (1 ); +return -EINVAL ; +} +static inline void kvm_gmem_handle_folio_put (struct folio *folio) +{ +WARN_ON_ONCE (1 ); +}#endif /* CONFIG_KVM_GMEM_MAPPABLE */ #endif
include/linux/page-flags.h +7 -0
@@ -942,6 +942,7 @@ enum pagetype { PGTY_slab =0xf5 , PGTY_zsmalloc =0xf6 , PGTY_unaccepted =0xf7 , + PGTY_guestmem =0xf8 , PGTY_mapcount_underflow = 0xff };
@@ -1091,6 +1092,12 @@ FOLIO_TYPE_OPS(hugetlb, hugetlb) FOLIO_TEST_FLAG_FALSE(hugetlb)#endif +#ifdef CONFIG_KVM_GMEM_MAPPABLE +FOLIO_TYPE_OPS(guestmem, guestmem) +#else +FOLIO_TEST_FLAG_FALSE(guestmem) +#endif +PAGE_TYPE_OPS (Zsmalloc, zsmalloc, zsmalloc)/*
mm/debug.c +1 -0
@@ -56,6 +56,7 @@ static const char *page_type_names[] = {DEF_PAGETYPE_NAME (table),DEF_PAGETYPE_NAME (buddy),DEF_PAGETYPE_NAME (unaccepted), + DEF_PAGETYPE_NAME(guestmem), };static const char *page_type_name (unsigned int page_type)
mm/swap.c +4 -0
@@ -37,6 +37,7 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/kvm_host.h> #include "internal.h"
@@ -103,6 +104,9 @@ static void free_typed_folio(struct folio *folio)case PGTY_offline:/* Nothing to do, it's offline. */ return ; +case PGTY_guestmem: +kvm_gmem_handle_folio_put (folio); +return ;default :WARN_ON_ONCE (1 ); }
virt/kvm/guest_memfd.c +145 -0
164 lines not on this page · raw message
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
kernel test robot
build botRe: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
kernel test robot noticed the following build errors:
Hi Fuad,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on 5bc55a333a2f7316b58edc7573e8e893f7acb532]
url: https://github.com/intel-lab-lkp/linux/commits/Fuad-Tabba/mm-Consolidate-freeing-of-typed-folios-on-final-folio_put/20250118-003538 base: 5bc55a333a2f7316b58edc7573e8e893f7acb532 patch link: https://lore.kernel.org/r/20250117163001.2326672-7-tabba%40google.com patch subject: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages config: arm-randconfig-001-20250118 (https://download.01.org/0day-ci/archive/20250118/[email protected]/config) compiler: arm-linux-gnueabi-gcc (GCC) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250118/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from include/linux/kvm_host.h:40,
from mm/swap.c:40:
>> include/uapi/linux/kvm.h:15:10: fatal error: asm/kvm.h: No such file or directory
15 | #include <asm/kvm.h>
| ^~~~~~~~~~~compilation terminated.
vim +15 include/uapi/linux/kvm.h
6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 4 6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 5 /* 6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 6 * Userspace interface for /dev/kvm - kernel based virtual machine 6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 7 * dea8caee7b6971 include/linux/kvm.h Rusty Russell 2007-07-17 8 * Note: you must update KVM_API_VERSION if you change this interface. 6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 9 */ 6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 10 fb1070d18edb37 include/uapi/linux/kvm.h Joe Richey 2021-05-21 11 #include <linux/const.h> 00bfddaf7f68a6 include/linux/kvm.h Jaswinder Singh Rajput 2009-01-15 12 #include <linux/types.h> 97646202bc3f19 include/linux/kvm.h Christian Borntraeger 2008-03-12 13 #include <linux/compiler.h> 6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 14 #include <linux/ioctl.h> f6a40e3bdf5fe0 include/linux/kvm.h Jerone Young 2007-11-19 @15 #include <asm/kvm.h> 6aa8b732ca01c3 include/linux/kvm.h Avi Kivity 2006-12-10 16
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Fuad TabbaQuoted 19 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
<snip>
Quoted 13 lines
virt/kvm/guest_memfd.c +7
-0@@ -387,6 +387,28 @@ enum folio_mappability { KVM_GMEM_NONE_MAPPABLE = 0b11, /* Not mappable, transient state. */ }; +/* +* Unregisters the __folio_put() callback from the folio. +* +* Restores a folio's refcount after all pending references have been released, +* and removes the folio type, thereby removing the callback. Now the folio can +* be freed normaly once all actual references have been dropped. +*
virt/kvm/guest_memfd.c enum folio_mappability { · line 406
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+* Must be called with the filemap (inode->i_mapping) invalidate_lock held. +* Must also have exclusive access to the folio: folio must be either locked, or +* gmem holds the only reference. +*/ +static void __kvm_gmem_restore_pending_folio (struct folio *folio) +{ +if (WARN_ON_ONCE (folio_mapped (folio) || !folio_test_guestmem (folio))) +return ; + +WARN_ON_ONCE (!folio_test_locked (folio) &&folio_ref_count (folio) >1 );
Similar to Kirill's objection on the other patch, I think there might be a speculative refcount increase (i.e. from a pfn scanner) as long as we have refcount over 1. Probably not a problem here if we want to restore refcount anyway? But the warning would be spurious.
Quoted 41 lines
virt/kvm/guest_memfd.c +29
-0+ +__folio_clear_guestmem (folio); +folio_ref_add (folio,folio_nr_pages (folio)); +} +/* * Marks the range [start, end) as mappable by both the host and the guest. * Usually called when guest shares memory with the host. @@ -400,7 +422,31 @@ static int gmem_set_mappable(struct inode *inode, pgoff_t start, pgoff_t end)filemap_invalidate_lock (inode->i_mapping);for (i = start; i < end; i++) { +struct folio *folio =NULL ; + +/* +* If the folio is NONE_MAPPABLE, it indicates that it is +* transitioning to private (GUEST_MAPPABLE). Transition it to +* shared (ALL_MAPPABLE) immediately, and remove the callback. +*/ +if (xa_to_value (xa_load (mappable_offsets, i)) ==KVM_GMEM_NONE_MAPPABLE ) { + folio =filemap_lock_folio (inode->i_mapping, i); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); +break ; + } + +if (folio_test_guestmem (folio)) +__kvm_gmem_restore_pending_folio (folio); + } + r =xa_err (xa_store (mappable_offsets, i, xval,GFP_KERNEL )); + +if (folio) { +folio_unlock (folio); +folio_put (folio); + } +if (r)break ; }
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 528
Quoted 10 lines
virt/kvm/guest_memfd.c +7
-0@@ -473,6 +519,105 @@ static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end)return r; } +/* +* Registers a callback to __folio_put(), so that gmem knows that the host does +* not have any references to the folio. It does that by setting the folio type +* to guestmem. +* +* Returns 0 if the host doesn't have any references, or -EAGAIN if the host +* has references, and the callback has been registered.
Note this comment.
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+* +* Must be called with the following locks held: +* - filemap (inode->i_mapping) invalidate_lock +* - folio lock +*/ +static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx)
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 544
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount; + +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ;
But here we return -EAGAIN and no callback was registered?
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+ +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 560
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ + + refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */
Again this seems racy, somebody could have just speculatively increased it. Maybe we need to freeze here as well?
Quoted 63 lines
virt/kvm/guest_memfd.c +60
-0+ r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Hi Vlastimil,
Quoted 117 lines · Vlastimil Babka
Quoted 19 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
<snip>
Quoted 13 lines
virt/kvm/guest_memfd.c +7
-0@@ -387,6 +387,28 @@ enum folio_mappability { KVM_GMEM_NONE_MAPPABLE = 0b11, /* Not mappable, transient state. */ }; +/* +* Unregisters the __folio_put() callback from the folio. +* +* Restores a folio's refcount after all pending references have been released, +* and removes the folio type, thereby removing the callback. Now the folio can +* be freed normaly once all actual references have been dropped. +* virt/kvm/guest_memfd.c enum folio_mappability { · line 406
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+* Must be called with the filemap (inode->i_mapping) invalidate_lock held. +* Must also have exclusive access to the folio: folio must be either locked, or +* gmem holds the only reference. +*/ +static void __kvm_gmem_restore_pending_folio (struct folio *folio) +{ +if (WARN_ON_ONCE (folio_mapped (folio) || !folio_test_guestmem (folio))) +return ; + +WARN_ON_ONCE (!folio_test_locked (folio) &&folio_ref_count (folio) >1 );Similar to Kirill's objection on the other patch, I think there might be a speculative refcount increase (i.e. from a pfn scanner) as long as we have refcount over 1. Probably not a problem here if we want to restore refcount anyway? But the warning would be spurious.
Quoted 41 lines
virt/kvm/guest_memfd.c +29
-0+ +__folio_clear_guestmem (folio); +folio_ref_add (folio,folio_nr_pages (folio)); +} +/* * Marks the range [start, end) as mappable by both the host and the guest. * Usually called when guest shares memory with the host. @@ -400,7 +422,31 @@ static int gmem_set_mappable(struct inode *inode, pgoff_t start, pgoff_t end)filemap_invalidate_lock (inode->i_mapping);for (i = start; i < end; i++) { +struct folio *folio =NULL ; + +/* +* If the folio is NONE_MAPPABLE, it indicates that it is +* transitioning to private (GUEST_MAPPABLE). Transition it to +* shared (ALL_MAPPABLE) immediately, and remove the callback. +*/ +if (xa_to_value (xa_load (mappable_offsets, i)) ==KVM_GMEM_NONE_MAPPABLE ) { + folio =filemap_lock_folio (inode->i_mapping, i); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); +break ; + } + +if (folio_test_guestmem (folio)) +__kvm_gmem_restore_pending_folio (folio); + } + r =xa_err (xa_store (mappable_offsets, i, xval,GFP_KERNEL )); + +if (folio) { +folio_unlock (folio); +folio_put (folio); + } +if (r)break ; }virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 528
Quoted 10 lines
virt/kvm/guest_memfd.c +7
-0@@ -473,6 +519,105 @@ static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end)return r; } +/* +* Registers a callback to __folio_put(), so that gmem knows that the host does +* not have any references to the folio. It does that by setting the folio type +* to guestmem. +* +* Returns 0 if the host doesn't have any references, or -EAGAIN if the host +* has references, and the callback has been registered. Note this comment.
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+* +* Must be called with the following locks held: +* - filemap (inode->i_mapping) invalidate_lock +* - folio lock +*/ +static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;
Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 544
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+ +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ;But here we return
-EAGAINand no callback was registered?
This is intentional. If the folio is still mapped (i.e., its mapcount is elevated), then we cannot register the callback yet, so the host/vmm needs to unmap first, then try again. That said, I see the problem with the comment above, and I will clarify this.
Quoted 11 lines
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ +
Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 560
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ Again this seems racy, somebody could have just speculatively increased it. Maybe we need to freeze here as well?
A speculative increase here is ok I think (famous last words). The callback was registered before the check, therefore, such an increase would trigger the callback.
Thanks,
/fuad
Quoted 64 lines
Quoted 63 lines
virt/kvm/guest_memfd.c +60
-0+ r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 56 lines · Fuad Tabba
Quoted 22 lines
Quoted 9 lines
<snip>
+/*
+ * Registers a callback to__folio_put(), so that gmem knows that the host does
+ * not have any references to the folio. It does that by setting the folio type
+ * to guestmem.
+ *
+ * Returns 0 if the host doesn't have any references, or-EAGAINif the host
+ * has references, and the callback has been registered.Note this comment.
Quoted 10 lines
+ *
+ * Must be called with the following locks held:
+ * - filemap (inode->i_mapping)invalidate_lock
+ * - folio lock
+ */virt/kvm/guest_memfd.c +5
-0+static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 544
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+ +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ;But here we return
-EAGAINand no callback was registered?This is intentional. If the folio is still mapped (i.e., its mapcount is elevated), then we cannot register the callback yet, so the host/vmm needs to unmap first, then try again. That said, I see the problem with the comment above, and I will clarify this.
Quoted 11 lines
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ +Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 560
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ Again this seems racy, somebody could have just speculatively increased it. Maybe we need to freeze here as well?
Quoted 8 lines
A speculative increase here is ok I think (famous last words). The callback was registered before the check, therefore, such an increase would trigger the callback.
Thanks,
/fuad
I checked the callback (kvm_gmem_handle_folio_put()) and agree with you that the mappability reset to KVM_GMEM_GUEST_MAPPABLE is handled correctly (since kvm_gmem_handle_folio_put() doesn't assume anything about the mappability state at callback-time).
However, what if the new speculative reference writes to the page and guest goes on to fault/use the page?
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines
virt/kvm/guest_memfd.c +60
-0+ r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 68 lines · Ackerley Tng
Quoted 56 lines · Fuad Tabba
Quoted 22 lines
Quoted 9 lines
<snip>
+/*
+ * Registers a callback to__folio_put(), so that gmem knows that the host does
+ * not have any references to the folio. It does that by setting the folio type
+ * to guestmem.
+ *
+ * Returns 0 if the host doesn't have any references, or-EAGAINif the host
+ * has references, and the callback has been registered.Note this comment.
Quoted 10 lines
+ *
+ * Must be called with the following locks held:
+ * - filemap (inode->i_mapping)invalidate_lock
+ * - folio lock
+ */virt/kvm/guest_memfd.c +5
-0+static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 544
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+ +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ;But here we return
-EAGAINand no callback was registered?This is intentional. If the folio is still mapped (i.e., its mapcount is elevated), then we cannot register the callback yet, so the host/vmm needs to unmap first, then try again. That said, I see the problem with the comment above, and I will clarify this.
Quoted 11 lines
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ +Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 560
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ Again this seems racy, somebody could have just speculatively increased it. Maybe we need to freeze here as well?
Quoted 8 lines
A speculative increase here is ok I think (famous last words). The callback was registered before the check, therefore, such an increase would trigger the callback.
Thanks,
/fuad
Quoted 7 lines
I checked the callback (
kvm_gmem_handle_folio_put()) and agree with you that the mappability reset toKVM_GMEM_GUEST_MAPPABLEis handled correctly (sincekvm_gmem_handle_folio_put()doesn't assume anything about the mappability state at callback-time).However, what if the new speculative reference writes to the page and guest goes on to fault/use the page?
I don't think that's a problem. At this point the page is in a transient state, but still shared from the guest's point of view. Moreover, no one can fault-in the page at the host at this point (we check in kvm_gmem_fault()).
Let's have a look at the code:
+static int __gmem_register_callback(struct folio *folio, struct inode *inode, pgoff_t idx)
+{
+ struct xarray *mappable_offsets =
&kvm_gmem_private(inode)->mappable_offsets;
+ void *xval_guest = xa_mk_value(KVM_GMEM_GUEST_MAPPABLE);
+ int refcount;
At this point the guest still perceives the page as shared, the state of the page is KVM_GMEM_NONE_MAPPABLE (transient state). This means that kvm_gmem_fault() doesn't fault-in the page at the host anymore.
+ rwsem_assert_held_write_nolockdep(&inode->i_mapping->invalidate_lock);
+ WARN_ON_ONCE(!folio_test_locked(folio));
+
+ if (folio_mapped(folio) || folio_test_guestmem(folio))
+ return -EAGAIN;
+
+ /* Register a callback first. */
+ __folio_set_guestmem(folio);
This (in addition to the state of the NONE_MAPPABLE), also ensures that kvm_gmem_fault() doesn't fault-in the page at the host anymore.
+ /*
+ * Check for references after setting the type to guestmem, to guard + * against potential races with the refcount being decremented later. + * + * At least one reference is expected because the folio is locked. + */
+
+ refcount = folio_ref_sub_return(folio, folio_nr_pages(folio));
+ if (refcount == 1) {
+ int r;
At this point we know that guest_memfd has the only real reference. Speculative references AFAIK do not access the page itself. +
+ /* refcount isn't elevated, it's now faultable by the guest. */
+ r = WARN_ON_ONCE(xa_err(xa_store(mappable_offsets,
idx, xval_guest, GFP_KERNEL)));
Now it's safe so let the guest know that it can map the page.
+ if (!r)
+ __kvm_gmem_restore_pending_folio(folio);
+
+ return r;
+ }
+
+ return -EAGAIN;
+}
Does this make sense, or did I miss something?
Thanks!
/fuad
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines
virt/kvm/guest_memfd.c +60
-0+ r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Fuad TabbaQuoted 137 lines · Fuad Tabba
Quoted 68 lines · Ackerley Tng
Quoted 56 lines · Fuad Tabba
Quoted 22 lines
Quoted 9 lines
<snip>
+/*
+ * Registers a callback to__folio_put(), so that gmem knows that the host does
+ * not have any references to the folio. It does that by setting the folio type
+ * to guestmem.
+ *
+ * Returns 0 if the host doesn't have any references, or-EAGAINif the host
+ * has references, and the callback has been registered.Note this comment.
Quoted 10 lines
+ *
+ * Must be called with the following locks held:
+ * - filemap (inode->i_mapping)invalidate_lock
+ * - folio lock
+ */virt/kvm/guest_memfd.c +5
-0+static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 544
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+ +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ;But here we return
-EAGAINand no callback was registered?This is intentional. If the folio is still mapped (i.e., its mapcount is elevated), then we cannot register the callback yet, so the host/vmm needs to unmap first, then try again. That said, I see the problem with the comment above, and I will clarify this.
Quoted 11 lines
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ +Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 560
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ Again this seems racy, somebody could have just speculatively increased it. Maybe we need to freeze here as well?
Quoted 8 lines
A speculative increase here is ok I think (famous last words). The callback was registered before the check, therefore, such an increase would trigger the callback.
Thanks,
/fuadQuoted 7 lines
I checked the callback (
kvm_gmem_handle_folio_put()) and agree with you that the mappability reset toKVM_GMEM_GUEST_MAPPABLEis handled correctly (sincekvm_gmem_handle_folio_put()doesn't assume anything about the mappability state at callback-time).However, what if the new speculative reference writes to the page and guest goes on to fault/use the page?
I don't think that's a problem. At this point the page is in a transient state, but still shared from the guest's point of view. Moreover, no one can fault-in the page at the host at this point (we check in
kvm_gmem_fault()).Let's have a look at the code:
+static int
__gmem_register_callback(struct folio *folio, struct inode *inode, pgoff_t idx)(quoted diff) +2
-0+{ + struct xarray *mappable_offsets =&kvm_gmem_private(inode)->mappable_offsets;
virt/kvm/guest_memfd.c +2
-0+void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;At this point the guest still perceives the page as shared, the state of the page is
KVM_GMEM_NONE_MAPPABLE(transient state). This means thatkvm_gmem_fault()doesn't fault-in the page at the host anymore.virt/kvm/guest_memfd.c +8
-0+rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ; + +/* Register a callback first. */ +__folio_set_guestmem (folio);This (in addition to the state of the
NONE_MAPPABLE), also ensures thatkvm_gmem_fault()doesn't fault-in the page at the host anymore.virt/kvm/guest_memfd.c +10
-0+/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ + + refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r;At this point we know that
guest_memfdhas the only real reference. Speculative references AFAIK do not access the page itself.virt/kvm/guest_memfd.c +3
-0+ +/* refcount isn't elevated, it's now faultable by the guest. */ + r = WARN_ON_ONCE(xa_err(xa_store(mappable_offsets,idx,
xval_guest,GFP_KERNEL)));Now it's safe so let the guest know that it can map the page.
virt/kvm/guest_memfd.c +8
-0+if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +}
Quoted 1 line
Does this make sense, or did I miss something?
Thanks for explaining! I don't know enough to confirm/deny this but I agree that if speculative references don't access the page itself, this works.
What if over here, we just drop the refcount, and let setting mappability to GUEST happen in the folio_put() callback?
Quoted 68 lines
Thanks!
/fuadQuoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines
virt/kvm/guest_memfd.c +60
-0+ r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 140 lines · Ackerley Tng
Quoted 137 lines · Fuad Tabba
Quoted 68 lines · Ackerley Tng
Quoted 56 lines · Fuad Tabba
Quoted 22 lines
Quoted 9 lines
<snip>
+/*
+ * Registers a callback to__folio_put(), so that gmem knows that the host does
+ * not have any references to the folio. It does that by setting the folio type
+ * to guestmem.
+ *
+ * Returns 0 if the host doesn't have any references, or-EAGAINif the host
+ * has references, and the callback has been registered.Note this comment.
Quoted 10 lines
+ *
+ * Must be called with the following locks held:
+ * - filemap (inode->i_mapping)invalidate_lock
+ * - folio lock
+ */virt/kvm/guest_memfd.c +5
-0+static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 544
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+ +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ;But here we return
-EAGAINand no callback was registered?This is intentional. If the folio is still mapped (i.e., its mapcount is elevated), then we cannot register the callback yet, so the host/vmm needs to unmap first, then try again. That said, I see the problem with the comment above, and I will clarify this.
Quoted 11 lines
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ +Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 560
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ Again this seems racy, somebody could have just speculatively increased it. Maybe we need to freeze here as well?
Quoted 8 lines
A speculative increase here is ok I think (famous last words). The callback was registered before the check, therefore, such an increase would trigger the callback.
Thanks,
/fuadQuoted 7 lines
I checked the callback (
kvm_gmem_handle_folio_put()) and agree with you that the mappability reset toKVM_GMEM_GUEST_MAPPABLEis handled correctly (sincekvm_gmem_handle_folio_put()doesn't assume anything about the mappability state at callback-time).However, what if the new speculative reference writes to the page and guest goes on to fault/use the page?
I don't think that's a problem. At this point the page is in a transient state, but still shared from the guest's point of view. Moreover, no one can fault-in the page at the host at this point (we check in
kvm_gmem_fault()).Let's have a look at the code:
+static int
__gmem_register_callback(struct folio *folio, struct inode *inode, pgoff_t idx)(quoted diff) +2
-0+{ + struct xarray *mappable_offsets =&kvm_gmem_private(inode)->mappable_offsets;
virt/kvm/guest_memfd.c +2
-0+void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;At this point the guest still perceives the page as shared, the state of the page is
KVM_GMEM_NONE_MAPPABLE(transient state). This means thatkvm_gmem_fault()doesn't fault-in the page at the host anymore.virt/kvm/guest_memfd.c +8
-0+rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ; + +/* Register a callback first. */ +__folio_set_guestmem (folio);This (in addition to the state of the
NONE_MAPPABLE), also ensures thatkvm_gmem_fault()doesn't fault-in the page at the host anymore.virt/kvm/guest_memfd.c +10
-0+/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ + + refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r;At this point we know that
guest_memfdhas the only real reference. Speculative references AFAIK do not access the page itself.virt/kvm/guest_memfd.c +3
-0+ +/* refcount isn't elevated, it's now faultable by the guest. */ + r = WARN_ON_ONCE(xa_err(xa_store(mappable_offsets,idx,
xval_guest,GFP_KERNEL)));Now it's safe so let the guest know that it can map the page.
virt/kvm/guest_memfd.c +8
-0+if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +}
Quoted 7 lines
Quoted 1 line
Does this make sense, or did I miss something?
Thanks for explaining! I don't know enough to confirm/deny this but I agree that if speculative references don't access the page itself, this works.
What if over here, we just drop the refcount, and let setting mappability to GUEST happen in the
folio_put()callback?
Similar to what I mentioned in the other thread, the common case should be that the mapcount and refcount are not elevated, therefore, I think it's better not to go through the callback route unless it's necessary for correctness.
Cheers,
/fuad
Quoted 68 lines
Quoted 68 lines
Thanks!
/fuadQuoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines
virt/kvm/guest_memfd.c +60
-0+ r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Ackerley TngHi Ackerley,
Quoted 68 lines · Ackerley Tng
Quoted 56 lines · Fuad Tabba
Quoted 22 lines
Quoted 9 lines
<snip>
+/*
+ * Registers a callback to__folio_put(), so that gmem knows that the host does
+ * not have any references to the folio. It does that by setting the folio type
+ * to guestmem.
+ *
+ * Returns 0 if the host doesn't have any references, or-EAGAINif the host
+ * has references, and the callback has been registered.Note this comment.
Quoted 10 lines
+ *
+ * Must be called with the following locks held:
+ * - filemap (inode->i_mapping)invalidate_lock
+ * - folio lock
+ */virt/kvm/guest_memfd.c +5
-0+static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount;Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 544
Quoted 6 lines
virt/kvm/guest_memfd.c +6
-0+ +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ;But here we return
-EAGAINand no callback was registered?This is intentional. If the folio is still mapped (i.e., its mapcount is elevated), then we cannot register the callback yet, so the host/vmm needs to unmap first, then try again. That said, I see the problem with the comment above, and I will clarify this.
Quoted 11 lines
Quoted 11 lines
virt/kvm/guest_memfd.c +11
-0+ +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ +Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 560
Quoted 5 lines
virt/kvm/guest_memfd.c +5
-0+ refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ Again this seems racy, somebody could have just speculatively increased it. Maybe we need to freeze here as well?
Quoted 8 lines
A speculative increase here is ok I think (famous last words). The callback was registered before the check, therefore, such an increase would trigger the callback.
Thanks,
/fuad
Quoted 7 lines
I checked the callback (
kvm_gmem_handle_folio_put()) and agree with you that the mappability reset toKVM_GMEM_GUEST_MAPPABLEis handled correctly (sincekvm_gmem_handle_folio_put()doesn't assume anything about the mappability state at callback-time).However, what if the new speculative reference writes to the page and guest goes on to fault/use the page?
In my last email I explained why I thought the code was fine as it is. Now that I'm updating the patch series with all the comments, I realized that even if I were right (which I am starting to doubt), freezing the refcount makes the code easier to reason about. So I'm going with ref_freeze here as well when I respin.
Thanks again,
/fuad
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines
virt/kvm/guest_memfd.c +60
-0+ r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Fuad TabbaFuad Tabba <[email protected]> writes:
Hey Fuad, I'm still working on verifying all this but for now this is one issue. I think this can be fixed by checking if the folio->mapping is NULL. If it's NULL, then the folio has been disassociated from the inode, and during the dissociation (removal from filemap), the mappability can also either
1. Be unset so that the default mappability can be set up based on GUEST_MEMFD_FLAG_INIT_MAPPABLE, or
2. Be directly restored based on GUEST_MEMFD_FLAG_INIT_MAPPABLE
Quoted 8 lines
<snip>
virt/kvm/guest_memfd.c +6
-0+ +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 610
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host;
IIUC this will be a NULL pointer dereference if the folio had been removed from the filemap, either through truncation or if the guest_memfd file got closed.
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 611
Quoted 1 line
virt/kvm/guest_memfd.c +1
-0+ index = folio->index;
And if removed from the filemap folio->index is probably invalid.
Quoted 12 lines
virt/kvm/guest_memfd.c +9
-0+ mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 9 lines · Ackerley Tng
Fuad Tabba <[email protected]> writes:
Hey Fuad, I'm still working on verifying all this but for now this is one issue. I think this can be fixed by checking if the
folio->mappingis NULL. If it's NULL, then the folio has been disassociated from the inode, and during the dissociation (removal from filemap), the mappability can also either
Quoted 3 lines
1. Be unset so that the default mappability can be set up based on
GUEST_MEMFD_FLAG_INIT_MAPPABLE, or
2. Be directly restored based onGUEST_MEMFD_FLAG_INIT_MAPPABLE
Thanks for pointing this out. I hadn't considered this case. I'll fix in the respin.
Quoted 14 lines
Quoted 14 lines
<snip>
virt/kvm/guest_memfd.c +12
-0+ +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode;
Quoted 8 lines
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 610
Quoted 4 lines
virt/kvm/guest_memfd.c +4
-0+pgoff_t index; +void *xval; + + inode = folio->mapping->host;IIUC this will be a NULL pointer dereference if the folio had been removed from the filemap, either through truncation or if the
guest_memfdfile got closed.
Ack.
Quoted 3 lines
Quoted 1 line
+ index =
folio->index;And if removed from the filemap
folio->indexis probably invalid.
Ack and thanks again,
/fuad
Quoted 12 lines
Quoted 12 lines
virt/kvm/guest_memfd.c +9
-0+ mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 11 lines · Fuad Tabba
Quoted 9 lines · Ackerley Tng
Fuad Tabba <[email protected]> writes:
Hey Fuad, I'm still working on verifying all this but for now this is one issue. I think this can be fixed by checking if the
folio->mappingis NULL. If it's NULL, then the folio has been disassociated from the inode, and during the dissociation (removal from filemap), the mappability can also either
Quoted 7 lines
Quoted 3 lines
1. Be unset so that the default mappability can be set up based on
GUEST_MEMFD_FLAG_INIT_MAPPABLE, or
2. Be directly restored based onGUEST_MEMFD_FLAG_INIT_MAPPABLEThanks for pointing this out. I hadn't considered this case. I'll fix in the respin.
Can the below scenario cause trouble?
1) Userspace converts a certain range of guest memfd as shared and grabs some refcounts on shared memory pages through existing kernel exposed mechanisms.
2) Userspace converts the same range to private which would cause the corresponding mappability attributes to be *MAPPABILITY_NONE.
3) Userspace truncates the range which will remove the page from pagecache.
4) Userspace does the fallocate again, leading to a new page getting allocated without freeing the older page which is still refcounted (step 1).
Effectively this could allow userspace to keep allocating multiple pages for the same guest_memfd range.
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 31 lines · Vishal Annapurve
Quoted 11 lines · Fuad Tabba
Quoted 9 lines · Ackerley Tng
Fuad Tabba <[email protected]> writes:
Hey Fuad, I'm still working on verifying all this but for now this is one issue. I think this can be fixed by checking if the
folio->mappingis NULL. If it's NULL, then the folio has been disassociated from the inode, and during the dissociation (removal from filemap), the mappability can also eitherQuoted 7 lines
Quoted 3 lines
1. Be unset so that the default mappability can be set up based on
GUEST_MEMFD_FLAG_INIT_MAPPABLE, or
2. Be directly restored based onGUEST_MEMFD_FLAG_INIT_MAPPABLEThanks for pointing this out. I hadn't considered this case. I'll fix in the respin.
Can the below scenario cause trouble?
1) Userspace converts a certain range of guest memfd as shared and grabs some refcounts on shared memory pages through existing kernel exposed mechanisms.
2) Userspace converts the same range to private which would cause the corresponding mappability attributes to be*MAPPABILITY_NONE.
3) Userspace truncates the range which will remove the page from pagecache.
4) Userspace does the fallocate again, leading to a new page getting allocated without freeing the older page which is still refcounted (step 1).
Quoted 2 lines
Effectively this could allow userspace to keep allocating multiple pages for the same
guest_memfdrange.
I'm still verifying this but for now here's the flow Vishal described in greater detail:
+ guest_memfd starts without GUEST_MEMFD_FLAG_INIT_MAPPABLE
+ All new pages will start with mappability = GUEST
+ guest uses a page
+ Get new page
+ Add page to filemap
+ guest converts page to shared
+ Mappability is now ALL
+ host uses page
+ host takes transient refcounts on page
+ Refcount on the page is now (a) filemap's refcount (b) vma's refcount
(c) transient refcount
+ guest converts page to private
+ Page is unmapped
+ Refcount on the page is now (a) filemap's refcount (b) transient refcount
+ Since refcount is elevated, the mappabilities are left as NONE
+ Filemap's refcounts are removed from the page
+ Refcount on the page is now (a) transient refcount
+ host punches hole to deallocate page
+ Since mappability was NONE, restore filemap's refcount
+ Refcount on the page is now (a) transient refcount (b) filemap's refcount
+ Mappabilities are reset to GUEST for truncated range
+ Folio is removed from filemap
+ Refcount on the page is now (a) transient refcount
+ Callback remains registered so that when the transient refcounts are dropped, cleanup can happen - this is where merging will happen with 1G page support
+ host fallocate()s in the same address range
+ will get a new page
Though the host does manage to get a new page while the old one stays around, I think this is working as intended, since the transient refcounts are truly holding the old folio around. When the transient refcounts go away, the old folio will still get cleaned up (with 1G page
support: merged and returned) to as expected. The new page will also be freed at some point later.
If the userspace program decides to keep taking transient refcounts to hold pages around, then the userspace program is truly leaking memory and it shouldn't be guest_memfd's bug.
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 78 lines · Ackerley Tng
Quoted 31 lines · Vishal Annapurve
Quoted 11 lines · Fuad Tabba
Quoted 9 lines · Ackerley Tng
Fuad Tabba <[email protected]> writes:
Hey Fuad, I'm still working on verifying all this but for now this is one issue. I think this can be fixed by checking if the
folio->mappingis NULL. If it's NULL, then the folio has been disassociated from the inode, and during the dissociation (removal from filemap), the mappability can also eitherQuoted 7 lines
Quoted 3 lines
1. Be unset so that the default mappability can be set up based on
GUEST_MEMFD_FLAG_INIT_MAPPABLE, or
2. Be directly restored based onGUEST_MEMFD_FLAG_INIT_MAPPABLEThanks for pointing this out. I hadn't considered this case. I'll fix in the respin.
Can the below scenario cause trouble?
1) Userspace converts a certain range of guest memfd as shared and grabs some refcounts on shared memory pages through existing kernel exposed mechanisms.
2) Userspace converts the same range to private which would cause the corresponding mappability attributes to be*MAPPABILITY_NONE.
3) Userspace truncates the range which will remove the page from pagecache.
4) Userspace does the fallocate again, leading to a new page getting allocated without freeing the older page which is still refcounted (step 1).Quoted 2 lines
Effectively this could allow userspace to keep allocating multiple pages for the same
guest_memfdrange.I'm still verifying this but for now here's the flow Vishal described in greater detail:
+
guest_memfdstarts withoutGUEST_MEMFD_FLAG_INIT_MAPPABLE
+ All new pages will start with mappability = GUEST
+ guest uses a page
+ Get new page
+ Add page to filemap
+ guest converts page to shared
+ Mappability is now ALL
+ host uses page
+ host takes transient refcounts on page
+ Refcount on the page is now (a) filemap's refcount (b) vma's refcount
(c) transient refcount
+ guest converts page to private
+ Page is unmapped
+ Refcount on the page is now (a) filemap's refcount (b) transient refcount
+ Since refcount is elevated, the mappabilities are left as NONE
+ Filemap's refcounts are removed from the page
+ Refcount on the page is now (a) transient refcount
+ host punches hole to deallocate page
+ Since mappability was NONE, restore filemap's refcount
+ Refcount on the page is now (a) transient refcount (b) filemap's refcount
+ Mappabilities are reset to GUEST for truncated range
+ Folio is removed from filemap
+ Refcount on the page is now (a) transient refcount
+ Callback remains registered so that when the transient refcounts are dropped, cleanup can happen - this is where merging will happen with 1G page support
+ host fallocate()s in the same address range
+ will get a new pageThough the host does manage to get a new page while the old one stays around, I think this is working as intended, since the transient refcounts are truly holding the old folio around. When the transient refcounts go away, the old folio will still get cleaned up (with 1G page
support: merged and returned) to as expected. The new page will also be freed at some point later.
Quoted 3 lines
If the userspace program decides to keep taking transient refcounts to hold pages around, then the userspace program is truly leaking memory and it shouldn't be guest_memfd's bug.
I wouldn't call such references transient. But a similar scenario is applicable for shmem files so it makes sense to call out this behavior as WAI.
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Fuad TabbaQuoted 12 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>
Quoted 8 lines
---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
Should the invocation of kvm_slot_gmem_register_callback() happen in the same critical block as setting the guest memfd range mappability to NONE, otherwise conversion/truncation could race with registration of callback?
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Hi Vishal,
Quoted 21 lines · Vishal Annapurve
Quoted 19 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.
Quoted 6 lines
Quoted 1 line
[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
Should the invocation of
kvm_slot_gmem_register_callback()happen in the same critical block as setting the guest memfd range mappability to NONE, otherwise conversion/truncation could race with registration of callback?
I don't think it needs to, at least not as far potencial races are concerned. First because kvm_slot_gmem_register_callback() grabs the mapping's invalidate_lock as well as the folio lock, and gmem_clear_mappable() grabs the mapping lock and the folio lock if a folio has been allocated before.
Second, __gmem_register_callback() checks before returning whether all references have been dropped, and adjusts the mappability/shareability if needed.
Cheers,
/fuad
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 32 lines · Fuad Tabba
Hi Vishal,
Quoted 21 lines · Vishal Annapurve
Quoted 19 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.Quoted 6 lines
Quoted 1 line
[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
Should the invocation of
kvm_slot_gmem_register_callback()happen in the same critical block as setting the guest memfd range mappability to NONE, otherwise conversion/truncation could race with registration of callback?
Quoted 5 lines
I don't think it needs to, at least not as far potencial races are concerned. First because
kvm_slot_gmem_register_callback()grabs the mapping'sinvalidate_lockas well as the folio lock, andgmem_clear_mappable()grabs the mapping lock and the folio lock if a folio has been allocated before.
I was hinting towards such a scenario:
Core1 Core2
Shared to private conversion ....
-> Results in mappability attributes
being set to NONE
...
Trigger private to shared conversion/truncation for
...
overlapping ranges
...kvm_slot_gmem_register_callback() on
the guest_memfd ranges converted
above (This will end up registering callback for guest_memfd ranges which possibly don't carry *_MAPPABILITY_NONE)
Quoted 7 lines
Second,
__gmem_register_callback()checks before returning whether all references have been dropped, and adjusts the mappability/shareability if needed.Cheers,
/fuad
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 40 lines · Vishal Annapurve
Quoted 32 lines · Fuad Tabba
Hi Vishal,
Quoted 21 lines · Vishal Annapurve
Quoted 19 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.Quoted 6 lines
Quoted 1 line
[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
Should the invocation of
kvm_slot_gmem_register_callback()happen in the same critical block as setting the guest memfd range mappability to NONE, otherwise conversion/truncation could race with registration of callback?Quoted 5 lines
I don't think it needs to, at least not as far potencial races are concerned. First because
kvm_slot_gmem_register_callback()grabs the mapping'sinvalidate_lockas well as the folio lock, andgmem_clear_mappable()grabs the mapping lock and the folio lock if a folio has been allocated before.
Quoted 16 lines
I was hinting towards such a scenario:
Core1
Shared to private conversion
-> Results in mappability attributes
being set to NONE
...
Trigger private to shared conversion/truncation for
...
overlapping ranges
...kvm_slot_gmem_register_callback()on
theguest_memfdranges converted
above (This will end up registering callback forguest_memfdranges which possibly don't carry*_MAPPABILITY_NONE)
Sorry for the format mess above.
I was hinting towards such a scenario:
Core1-
Shared to private conversion -> Results in mappability attributes being set to NONE
...
Core2
Trigger private to shared conversion/truncation for overlapping ranges
...
Core1kvm_slot_gmem_register_callback() on the guest_memfd ranges converted above (This will end up registering callback for guest_memfd ranges which possibly don't carry *_MAPPABILITY_NONE)
Quoted 7 lines
Quoted 7 lines
Second,
__gmem_register_callback()checks before returning whether all references have been dropped, and adjusts the mappability/shareability if needed.Cheers,
/fuad
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 60 lines · Vishal Annapurve
Quoted 40 lines · Vishal Annapurve
Quoted 32 lines · Fuad Tabba
Hi Vishal,
Quoted 21 lines · Vishal Annapurve
Quoted 19 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.Quoted 6 lines
Quoted 1 line
[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
Should the invocation of
kvm_slot_gmem_register_callback()happen in the same critical block as setting the guest memfd range mappability to NONE, otherwise conversion/truncation could race with registration of callback?Quoted 5 lines
I don't think it needs to, at least not as far potencial races are concerned. First because
kvm_slot_gmem_register_callback()grabs the mapping'sinvalidate_lockas well as the folio lock, andgmem_clear_mappable()grabs the mapping lock and the folio lock if a folio has been allocated before.Quoted 16 lines
I was hinting towards such a scenario:
Core1
Shared to private conversion
-> Results in mappability attributes
being set to NONE
...
Trigger private to shared conversion/truncation for
...
overlapping ranges
...kvm_slot_gmem_register_callback()on
theguest_memfdranges converted
above (This will end up registering callback forguest_memfdranges which possibly don't carry*_MAPPABILITY_NONE)Sorry for the format mess above.
Quoted 13 lines
I was hinting towards such a scenario:
Core1-
Shared to private conversion -> Results in mappability attributes being set to NONE
...
Core2
Trigger private to shared conversion/truncation for overlapping ranges
...
Core1kvm_slot_gmem_register_callback()on theguest_memfdranges converted above (This will end up registering callback forguest_memfdranges which possibly don't carry*_MAPPABILITY_NONE)
In my model (I'm working through internal processes to open source this) I set up the the folio_put() callback to be registered on truncation regardless of mappability state.
The folio_put() callback has multiple purposes, see slide 5 of this deck [1]:
1. Transitioning mappability from NONE to GUEST
2. Merging the folio if it is ready for merging
3. Keeping subfolio around (even if refcount == 0) until folio is ready for merging or return it to hugetlb
So it is okay and in fact better to have the callback registered:
1. Folios with mappability == NONE can be transitioned to GUEST
2. Folios with mappability == GUEST/ALL can be merged if the other subfolios are ready for merging
3. And no matter the mappability, if subfolios are not yet merged, they have to be kept around even with refcount 0 until they are merged.
The model doesn't model locking so I'll have to code it up for real to verify this, but for now I think we should take a mappability lock during mappability read/write, and do any necessary callback (un)registration while holding the lock. There's no concern of nested locking here since callback registration will purely (un)set PGTY_guest_memfd and does not add/drop refcounts.
With the callback registration locked with mappability updates, the refcounting and folio_put() callback should keep guest_memfd in a consistent state.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Second,
__gmem_register_callback()checks before returning whether all references have been dropped, and adjusts the mappability/shareability if needed.Cheers,
/fuad
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Hi Ackerley,
Quoted 104 lines · Ackerley Tng
Quoted 60 lines · Vishal Annapurve
Quoted 40 lines · Vishal Annapurve
Quoted 32 lines · Fuad Tabba
Hi Vishal,
Quoted 21 lines · Vishal Annapurve
Quoted 19 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.Quoted 6 lines
Quoted 1 line
[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
Should the invocation of
kvm_slot_gmem_register_callback()happen in the same critical block as setting the guest memfd range mappability to NONE, otherwise conversion/truncation could race with registration of callback?Quoted 5 lines
I don't think it needs to, at least not as far potencial races are concerned. First because
kvm_slot_gmem_register_callback()grabs the mapping'sinvalidate_lockas well as the folio lock, andgmem_clear_mappable()grabs the mapping lock and the folio lock if a folio has been allocated before.Quoted 16 lines
I was hinting towards such a scenario:
Core1
Shared to private conversion
-> Results in mappability attributes
being set to NONE
...
Trigger private to shared conversion/truncation for
...
overlapping ranges
...kvm_slot_gmem_register_callback()on
theguest_memfdranges converted
above (This will end up registering callback forguest_memfdranges which possibly don't carry*_MAPPABILITY_NONE)Sorry for the format mess above.
Quoted 13 lines
I was hinting towards such a scenario:
Core1-
Shared to private conversion -> Results in mappability attributes being set to NONE
...
Core2
Trigger private to shared conversion/truncation for overlapping ranges
...
Core1kvm_slot_gmem_register_callback()on theguest_memfdranges converted above (This will end up registering callback forguest_memfdranges which possibly don't carry*_MAPPABILITY_NONE)In my model (I'm working through internal processes to open source this) I set up the the
folio_put()callback to be registered on truncation regardless of mappability state.The
folio_put()callback has multiple purposes, see slide 5 of this deck [1]:1. Transitioning mappability from NONE to GUEST
2. Merging the folio if it is ready for merging
3. Keeping subfolio around (even if refcount == 0) until folio is ready for merging or return it to hugetlbSo it is okay and in fact better to have the callback registered:
1. Folios with mappability == NONE can be transitioned to GUEST
2. Folios with mappability == GUEST/ALL can be merged if the other subfolios are ready for merging
3. And no matter the mappability, if subfolios are not yet merged, they have to be kept around even with refcount 0 until they are merged.The model doesn't model locking so I'll have to code it up for real to verify this, but for now I think we should take a mappability lock during mappability read/write, and do any necessary callback (un)registration while holding the lock. There's no concern of nested locking here since callback registration will purely (un)set
PGTY_guest_memfdand does not add/drop refcounts.
Quoted 3 lines
With the callback registration locked with mappability updates, the refcounting and
folio_put()callback should keepguest_memfdin a consistent state.
So if I understand you correctly, we'll need to always register for large folios, right? If that's the case, we could expand the check to whether to register the callback, and ensure it's always registered for large folios. Since, like I said, the common case for small folios is that it would be just additional overhead. Right?
Cheers,
/fuad
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Second,
__gmem_register_callback()checks before returning whether all references have been dropped, and adjusts the mappability/shareability if needed.Cheers,
/fuad
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Fuad TabbaQuoted 1 line · Fuad Tabba
<snip>
Quoted 15 lines
static const char *page_type_name(unsigned int
page_type)mm/swap.c +1
-0@@ -37,6 +37,7 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/kvm_host.h> #include "internal.h"
mm/swap.c static void free_typed_folio(struct folio *folio) · line 109
Quoted 6 lines
mm/swap.c +3
-0@@ -103,6 +104,9 @@ static void free_typed_folio(struct folio *folio)case PGTY_offline:/* Nothing to do, it's offline. */ return ; +case PGTY_guestmem: +kvm_gmem_handle_folio_put (folio); +return ;
Unless it's discussed before, kvm_gmem_handle_folio_put() needs to be implemented outside KVM code which could be unloaded at runtime. Eliott's plan [1] to implement a guest_memfd library can handle this scenario in future.
[1] https://patches.linaro.org/project/linux-arm-msm/patch/[email protected]/
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Hi Vishal,
Quoted 31 lines · Vishal Annapurve
Quoted 1 line · Fuad Tabba
<snip>
Quoted 15 lines
static const char *page_type_name(unsigned int
page_type)mm/swap.c +1
-0@@ -37,6 +37,7 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/kvm_host.h> #include "internal.h" mm/swap.c static void free_typed_folio(struct folio *folio) · line 109
Quoted 6 lines
mm/swap.c +3
-0@@ -103,6 +104,9 @@ static void free_typed_folio(struct folio *folio)case PGTY_offline:/* Nothing to do, it's offline. */ return ; +case PGTY_guestmem: +kvm_gmem_handle_folio_put (folio); +return ;Unless it's discussed before,
kvm_gmem_handle_folio_put()needs to be implemented outside KVM code which could be unloaded at runtime. Eliott's plan [1] to implement aguest_memfdlibrary can handle this scenario in future.
Yes, not just that, but there's a lot of KVM code in guest_memdf in general.
Cheers,
/fuad
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
In reply to Fuad TabbaQuoted 181 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
--- include/linux/kvm_host.h | 11 +++ include/linux/page-flags.h | 7 ++ mm/debug.c | 1 + mm/swap.c | 4 + virt/kvm/guest_memfd.c | 145 +++++++++++++++++++++++++++++++++++++ 5 files changed, 168 insertions(+)include/linux/kvm_host.h +11
-0@@ -2574,6 +2574,8 @@ int kvm_slot_gmem_clear_mappable(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);bool kvm_slot_gmem_is_mappable (struct kvm_memory_slot *slot,gfn_t gfn);bool kvm_slot_gmem_is_guest_mappable (struct kvm_memory_slot *slot,gfn_t gfn); +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn); +void kvm_gmem_handle_folio_put (struct folio *folio);#else static inline bool kvm_gmem_is_mappable (struct kvm *kvm,gfn_t gfn,gfn_t end) {@@ -2615,6 +2617,15 @@ static inline bool kvm_slot_gmem_is_guest_mappable(struct kvm_memory_slot *slot,WARN_ON_ONCE (1 );return false; } +static inline int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +WARN_ON_ONCE (1 ); +return -EINVAL ; +} +static inline void kvm_gmem_handle_folio_put (struct folio *folio) +{ +WARN_ON_ONCE (1 ); +}#endif /* CONFIG_KVM_GMEM_MAPPABLE */ #endif include/linux/page-flags.h +7
-0@@ -942,6 +942,7 @@ enum pagetype { PGTY_slab =0xf5 , PGTY_zsmalloc =0xf6 , PGTY_unaccepted =0xf7 , + PGTY_guestmem =0xf8 , PGTY_mapcount_underflow = 0xff };@@ -1091,6 +1092,12 @@ FOLIO_TYPE_OPS(hugetlb, hugetlb) FOLIO_TEST_FLAG_FALSE(hugetlb)#endif +#ifdef CONFIG_KVM_GMEM_MAPPABLE +FOLIO_TYPE_OPS(guestmem, guestmem) +#else +FOLIO_TEST_FLAG_FALSE(guestmem) +#endif +PAGE_TYPE_OPS (Zsmalloc, zsmalloc, zsmalloc)/* mm/debug.c +1
-0@@ -56,6 +56,7 @@ static const char *page_type_names[] = {DEF_PAGETYPE_NAME (table),DEF_PAGETYPE_NAME (buddy),DEF_PAGETYPE_NAME (unaccepted), + DEF_PAGETYPE_NAME(guestmem), };static const char *page_type_name (unsigned int page_type)mm/swap.c +4
-0@@ -37,6 +37,7 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/kvm_host.h> #include "internal.h" @@ -103,6 +104,9 @@ static void free_typed_folio(struct folio *folio)case PGTY_offline:/* Nothing to do, it's offline. */ return ; +case PGTY_guestmem: +kvm_gmem_handle_folio_put (folio); +return ;default :WARN_ON_ONCE (1 ); }virt/kvm/guest_memfd.c +46
-0@@ -387,6 +387,28 @@ enum folio_mappability { KVM_GMEM_NONE_MAPPABLE = 0b11, /* Not mappable, transient state. */ }; +/* +* Unregisters the __folio_put() callback from the folio. +* +* Restores a folio's refcount after all pending references have been released, +* and removes the folio type, thereby removing the callback. Now the folio can +* be freed normaly once all actual references have been dropped. +* +* Must be called with the filemap (inode->i_mapping) invalidate_lock held. +* Must also have exclusive access to the folio: folio must be either locked, or +* gmem holds the only reference. +*/ +static void __kvm_gmem_restore_pending_folio (struct folio *folio) +{ +if (WARN_ON_ONCE (folio_mapped (folio) || !folio_test_guestmem (folio))) +return ; + +WARN_ON_ONCE (!folio_test_locked (folio) &&folio_ref_count (folio) >1 ); + +__folio_clear_guestmem (folio); +folio_ref_add (folio,folio_nr_pages (folio)); +} +/* * Marks the range [start, end) as mappable by both the host and the guest. * Usually called when guest shares memory with the host. @@ -400,7 +422,31 @@ static int gmem_set_mappable(struct inode *inode, pgoff_t start, pgoff_t end)filemap_invalidate_lock (inode->i_mapping);for (i = start; i < end; i++) { +struct folio *folio =NULL ; + +/* +* If the folio is NONE_MAPPABLE, it indicates that it is +* transitioning to private (GUEST_MAPPABLE). Transition it to +* shared (ALL_MAPPABLE) immediately, and remove the callback. +*/ +if (xa_to_value (xa_load (mappable_offsets, i)) ==KVM_GMEM_NONE_MAPPABLE ) { + folio =filemap_lock_folio (inode->i_mapping, i); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); +break ; + } + +if (folio_test_guestmem (folio)) +__kvm_gmem_restore_pending_folio (folio); + } + r =xa_err (xa_store (mappable_offsets, i, xval,GFP_KERNEL )); + +if (folio) { +folio_unlock (folio); +folio_put (folio); + } +if (r)break ; }
virt/kvm/guest_memfd.c static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) · line 521
Quoted 3 lines
virt/kvm/guest_memfd.c +0
-0@@ -473,6 +519,105 @@ static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end)return r; }
I think one of these functions to restore mappability needs to be called to restore the refcounts on truncation. Without doing this, the refcounts on the folios at truncation time would only be the transient/speculative ones, and truncating will take off the filemap refcounts which were already taken off to set up the folio_put() callback.
Should mappability can be restored according to GUEST_MEMFD_FLAG_INIT_MAPPABLE? Or should mappability of NONE be restored to GUEST and mappability of ALL left as ALL?
Quoted 102 lines
virt/kvm/guest_memfd.c +99
-0+/* +* Registers a callback to __folio_put(), so that gmem knows that the host does +* not have any references to the folio. It does that by setting the folio type +* to guestmem. +* +* Returns 0 if the host doesn't have any references, or -EAGAIN if the host +* has references, and the callback has been registered. +* +* Must be called with the following locks held: +* - filemap (inode->i_mapping) invalidate_lock +* - folio lock +*/ +static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount; + +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ; + +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ + + refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ + r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages
Quoted 186 lines · Ackerley Tng
Quoted 183 lines · Fuad Tabba
Before transitioning a
guest_memfdfolio to unshared, thereby disallowing access by the host and allowing the hypervisor to transition its view of the guest page as private, we need to be sure that the host doesn't have any references to the folio.This patch introduces a new type for
guest_memfdfolios, and uses that to register a callback that informs theguest_memfdsubsystem when the last reference is dropped, therefore knowing that the host doesn't have any remaining references.Signed-off-by Fuad Tabba <[email protected]>---
The functionkvm_slot_gmem_register_callback()isn't used in this series. It will be used later in code that performs unsharing of memory. I have tested it with pKVM, based on downstream code [*]. It's included in this RFC since it demonstrates the plan to handle unsharing of private folios.[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
--- include/linux/kvm_host.h | 11 +++ include/linux/page-flags.h | 7 ++ mm/debug.c | 1 + mm/swap.c | 4 + virt/kvm/guest_memfd.c | 145 +++++++++++++++++++++++++++++++++++++ 5 files changed, 168 insertions(+)include/linux/kvm_host.h +11
-0@@ -2574,6 +2574,8 @@ int kvm_slot_gmem_clear_mappable(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);bool kvm_slot_gmem_is_mappable (struct kvm_memory_slot *slot,gfn_t gfn);bool kvm_slot_gmem_is_guest_mappable (struct kvm_memory_slot *slot,gfn_t gfn); +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn); +void kvm_gmem_handle_folio_put (struct folio *folio);#else static inline bool kvm_gmem_is_mappable (struct kvm *kvm,gfn_t gfn,gfn_t end) {@@ -2615,6 +2617,15 @@ static inline bool kvm_slot_gmem_is_guest_mappable(struct kvm_memory_slot *slot,WARN_ON_ONCE (1 );return false; } +static inline int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +WARN_ON_ONCE (1 ); +return -EINVAL ; +} +static inline void kvm_gmem_handle_folio_put (struct folio *folio) +{ +WARN_ON_ONCE (1 ); +}#endif /* CONFIG_KVM_GMEM_MAPPABLE */ #endif include/linux/page-flags.h +7
-0@@ -942,6 +942,7 @@ enum pagetype { PGTY_slab =0xf5 , PGTY_zsmalloc =0xf6 , PGTY_unaccepted =0xf7 , + PGTY_guestmem =0xf8 , PGTY_mapcount_underflow = 0xff };@@ -1091,6 +1092,12 @@ FOLIO_TYPE_OPS(hugetlb, hugetlb) FOLIO_TEST_FLAG_FALSE(hugetlb)#endif +#ifdef CONFIG_KVM_GMEM_MAPPABLE +FOLIO_TYPE_OPS(guestmem, guestmem) +#else +FOLIO_TEST_FLAG_FALSE(guestmem) +#endif +PAGE_TYPE_OPS (Zsmalloc, zsmalloc, zsmalloc)/* mm/debug.c +1
-0@@ -56,6 +56,7 @@ static const char *page_type_names[] = {DEF_PAGETYPE_NAME (table),DEF_PAGETYPE_NAME (buddy),DEF_PAGETYPE_NAME (unaccepted), + DEF_PAGETYPE_NAME(guestmem), };static const char *page_type_name (unsigned int page_type)mm/swap.c +4
-0@@ -37,6 +37,7 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/kvm_host.h> #include "internal.h" @@ -103,6 +104,9 @@ static void free_typed_folio(struct folio *folio)case PGTY_offline:/* Nothing to do, it's offline. */ return ; +case PGTY_guestmem: +kvm_gmem_handle_folio_put (folio); +return ;default :WARN_ON_ONCE (1 ); }virt/kvm/guest_memfd.c +46
-0@@ -387,6 +387,28 @@ enum folio_mappability { KVM_GMEM_NONE_MAPPABLE = 0b11, /* Not mappable, transient state. */ }; +/* +* Unregisters the __folio_put() callback from the folio. +* +* Restores a folio's refcount after all pending references have been released, +* and removes the folio type, thereby removing the callback. Now the folio can +* be freed normaly once all actual references have been dropped. +* +* Must be called with the filemap (inode->i_mapping) invalidate_lock held. +* Must also have exclusive access to the folio: folio must be either locked, or +* gmem holds the only reference. +*/ +static void __kvm_gmem_restore_pending_folio (struct folio *folio) +{ +if (WARN_ON_ONCE (folio_mapped (folio) || !folio_test_guestmem (folio))) +return ; + +WARN_ON_ONCE (!folio_test_locked (folio) &&folio_ref_count (folio) >1 ); + +__folio_clear_guestmem (folio); +folio_ref_add (folio,folio_nr_pages (folio)); +} +/* * Marks the range [start, end) as mappable by both the host and the guest. * Usually called when guest shares memory with the host. @@ -400,7 +422,31 @@ static int gmem_set_mappable(struct inode *inode, pgoff_t start, pgoff_t end)filemap_invalidate_lock (inode->i_mapping);for (i = start; i < end; i++) { +struct folio *folio =NULL ; + +/* +* If the folio is NONE_MAPPABLE, it indicates that it is +* transitioning to private (GUEST_MAPPABLE). Transition it to +* shared (ALL_MAPPABLE) immediately, and remove the callback. +*/ +if (xa_to_value (xa_load (mappable_offsets, i)) ==KVM_GMEM_NONE_MAPPABLE ) { + folio =filemap_lock_folio (inode->i_mapping, i); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); +break ; + } + +if (folio_test_guestmem (folio)) +__kvm_gmem_restore_pending_folio (folio); + } + r =xa_err (xa_store (mappable_offsets, i, xval,GFP_KERNEL )); + +if (folio) { +folio_unlock (folio); +folio_put (folio); + } +if (r)break ; }@@ -473,6 +519,105 @@ static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end)return r; }
Quoted 8 lines
Quoted 1 line
I think one of these functions to restore mappability needs to be called to restore the refcounts on truncation. Without doing this, the refcounts on the folios at truncation time would only be the transient/speculative ones, and truncating will take off the filemap refcounts which were already taken off to set up the
folio_put()callback.
Good point.
Quoted 3 lines
Should mappability can be restored according to
GUEST_MEMFD_FLAG_INIT_MAPPABLE? Or should mappability of NONE be restored to GUEST and mappability of ALL left as ALL?
Not sure I follow :)
Thanks,
/fuad
Quoted 102 lines
Quoted 102 lines
virt/kvm/guest_memfd.c +99
-0+/* +* Registers a callback to __folio_put(), so that gmem knows that the host does +* not have any references to the folio. It does that by setting the folio type +* to guestmem. +* +* Returns 0 if the host doesn't have any references, or -EAGAIN if the host +* has references, and the callback has been registered. +* +* Must be called with the following locks held: +* - filemap (inode->i_mapping) invalidate_lock +* - folio lock +*/ +static int __gmem_register_callback (struct folio *folio,struct inode *inode,pgoff_t idx) +{ +struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; +void *xval_guest =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); +int refcount; + +rwsem_assert_held_write_nolockdep (&inode->i_mapping->invalidate_lock); +WARN_ON_ONCE (!folio_test_locked (folio)); + +if (folio_mapped (folio) ||folio_test_guestmem (folio)) +return -EAGAIN ; + +/* Register a callback first. */ +__folio_set_guestmem (folio); + +/* +* Check for references after setting the type to guestmem, to guard +* against potential races with the refcount being decremented later. +* +* At least one reference is expected because the folio is locked. +*/ + + refcount =folio_ref_sub_return (folio,folio_nr_pages (folio)); +if (refcount ==1 ) { +int r; + +/* refcount isn't elevated, it's now faultable by the guest. */ + r =WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, idx, xval_guest,GFP_KERNEL ))); +if (!r) +__kvm_gmem_restore_pending_folio (folio); + +return r; + } + +return -EAGAIN ; +} + +int kvm_slot_gmem_register_callback (struct kvm_memory_slot *slot,gfn_t gfn) +{ +unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; +struct inode *inode =file_inode (slot->gmem.file); +struct folio *folio; +int r; + +filemap_invalidate_lock (inode->i_mapping); + + folio =filemap_lock_folio (inode->i_mapping, pgoff); +if (WARN_ON_ONCE (IS_ERR (folio))) { + r =PTR_ERR (folio); + goto out; + } + + r =__gmem_register_callback (folio, inode, pgoff); + +folio_unlock (folio); +folio_put (folio); +out: +filemap_invalidate_unlock (inode->i_mapping); + +return r; +} + +/* +* Callback function for __folio_put(), i.e., called when all references by the +* host to the folio have been dropped. This allows gmem to transition the state +* of the folio to mappable by the guest, and allows the hypervisor to continue +* transitioning its state to private, since the host cannot attempt to access +* it anymore. +*/ +void kvm_gmem_handle_folio_put (struct folio *folio) +{ +struct xarray *mappable_offsets; +struct inode *inode; +pgoff_t index; +void *xval; + + inode = folio->mapping->host; + index = folio->index; + mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets; + xval =xa_mk_value (KVM_GMEM_GUEST_MAPPABLE ); + +filemap_invalidate_lock (inode->i_mapping); +__kvm_gmem_restore_pending_folio (folio); +WARN_ON_ONCE (xa_err (xa_store (mappable_offsets, index, xval,GFP_KERNEL ))); +filemap_invalidate_unlock (inode->i_mapping); +} +static bool gmem_is_mappable (struct inode *inode,pgoff_t pgoff) {struct xarray *mappable_offsets = &kvm_gmem_private (inode)->mappable_offsets;
RFC PATCH v5 7/15 KVM: guest_memfd: Allow host to mmap guest_memfd() pages when shared
In reply to Fuad TabbaAdd support for mmap() and fault() for guest_memfd in the host. The ability to fault in a guest page is contingent on that page being shared with the host.
The guest_memfd PRIVATE memory attribute is not used for two reasons. First because it reflects the userspace expectation for that memory location, and therefore can be toggled by userspace. The second is, although each guest_memfd file has a 1:1 binding with a KVM instance, the plan is to allow multiple files per inode, e.g. to allow intra-host migration to a new KVM instance, without destroying guest_memfd.
The mapping is restricted to only memory explicitly shared with the host. KVM checks that the host doesn't have any mappings for private memory via the folio's refcount. To avoid races between paths that check mappability and paths that check whether the host has any mappings (via the refcount), the folio lock is held in while either check is being performed.
This new feature is gated with a new configuration option, CONFIG_KVM_GMEM_MAPPABLE.
---
The functions kvm_gmem_is_mapped(), kvm_gmem_set_mappable(), and int kvm_gmem_clear_mappable() are not used in this patch series. They are intended to be used in future patches [*], which check and toggle mapability when the guest shares/unshares pages with the host.
[*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm
--- virt/kvm/Kconfig | 4 ++ virt/kvm/guest_memfd.c | 87 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)
virt/kvm/Kconfig +4 -0
@@ -124,3 +124,7 @@ config HAVE_KVM_ARCH_GMEM_PREPARE config HAVE_KVM_ARCH_GMEM_INVALIDATE bool depends on KVM_PRIVATE_MEM + +config KVM_GMEM_MAPPABLE + select KVM_PRIVATE_MEM + bool
virt/kvm/guest_memfd.c +87 -0
102 lines not on this page · raw message
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 8/15 KVM: guest_memfd: Add guest_memfd support to kvm_(read|/write)_guest_page()
In reply to Fuad TabbaMake kvm_(read|/write)_guest_page() capable of accessing guest memory for slots that don't have a userspace address, but only if the memory is mappable, which also indicates that it is accessible by the host.
--- virt/kvm/kvm_main.c | 133 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 114 insertions(+), 19 deletions(-)
virt/kvm/kvm_main.c +114 -19
@@ -3184,23 +3184,110 @@ int kvm_gmem_clear_mappable(struct kvm *kvm, gfn_t start, gfn_t end)return r; } +static int __kvm_read_guest_memfd_page (struct kvm *kvm, +struct kvm_memory_slot *slot, +gfn_t gfn,void *data,int offset, +int len) +{ +struct page *page; +u64 pfn; +int r; + +/* +* Holds the folio lock until after checking whether it can be faulted +* in, to avoid races with paths that change a folio's mappability. +*/ + r =kvm_gmem_get_pfn_locked (kvm, slot, gfn, &pfn, &page,NULL ); +if (r) +return r; + +if (!kvm_gmem_is_mappable (kvm, gfn, gfn +1 )) { + r = -EPERM ; + goto unlock; + } +memcpy (data,page_address (page) + offset, len); +unlock: +unlock_page (page); +if (r) +put_page (page); +else +kvm_release_page_clean (page); + +return r; +} + +static int __kvm_write_guest_memfd_page (struct kvm *kvm, +struct kvm_memory_slot *slot, +gfn_t gfn,const void *data, +int offset,int len) +{ +struct page *page; +u64 pfn; +int r; + +/* +* Holds the folio lock until after checking whether it can be faulted +* in, to avoid races with paths that change a folio's mappability. +*/ + r =kvm_gmem_get_pfn_locked (kvm, slot, gfn, &pfn, &page,NULL ); +if (r) +return r; + +if (!kvm_gmem_is_mappable (kvm, gfn, gfn +1 )) { + r = -EPERM ; + goto unlock; + } +memcpy (page_address (page) + offset, data, len); +unlock: +unlock_page (page); +if (r) +put_page (page); +else +kvm_release_page_dirty (page); + +return r; +} +#else +static int __kvm_read_guest_memfd_page (struct kvm *kvm, +struct kvm_memory_slot *slot, +gfn_t gfn,void *data,int offset, +int len) +{ +WARN_ON_ONCE (1 ); +return -EIO ; +} + +static int __kvm_write_guest_memfd_page (struct kvm *kvm, +struct kvm_memory_slot *slot, +gfn_t gfn,const void *data, +int offset,int len) +{ +WARN_ON_ONCE (1 ); +return -EIO ; +}#endif /* CONFIG_KVM_GMEM_MAPPABLE */ /* Copy @len bytes from guest memory at '(@gfn * PAGE_SIZE) + @offset' to @data */ -static int __kvm_read_guest_page (struct kvm_memory_slot *slot,gfn_t gfn, -void *data,int offset,int len) + +static int __kvm_read_guest_page (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,void *data,int offset,int len) { -int r;unsigned long addr;if (WARN_ON_ONCE (offset + len >PAGE_SIZE ))return -EFAULT ; +if (IS_ENABLED (CONFIG_KVM_GMEM_MAPPABLE ) && +kvm_slot_can_be_private (slot) && + !slot->userspace_addr) { +return __kvm_read_guest_memfd_page (kvm, slot, gfn, data, + offset, len); + } + addr =gfn_to_hva_memslot_prot (slot, gfn,NULL );if (kvm_is_error_hva (addr))return -EFAULT ; - r =__copy_from_user (data, (void __user *)addr + offset, len); -if (r) +if (__copy_from_user (data, (void __user *)addr + offset, len))return -EFAULT ;return 0 ; }
@@ -3210,7 +3297,7 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset, {struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (kvm, slot, gfn, data, offset, len); }EXPORT_SYMBOL_GPL (kvm_read_guest_page);
@@ -3219,7 +3306,7 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, {struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (vcpu->kvm, slot, gfn, data, offset, len); }EXPORT_SYMBOL_GPL (kvm_vcpu_read_guest_page);
@@ -3296,22 +3383,30 @@ EXPORT_SYMBOL_GPL(kvm_vcpu_read_guest_atomic);/* Copy @len bytes from @data into guest memory at '(@gfn * PAGE_SIZE) + @offset' */ static int __kvm_write_guest_page (struct kvm *kvm, -struct kvm_memory_slot *memslot,gfn_t gfn, -const void *data,int offset,int len) +struct kvm_memory_slot *slot,gfn_t gfn, +const void *data,int offset,int len) { -int r; -unsigned long addr; -if (WARN_ON_ONCE (offset + len >PAGE_SIZE ))return -EFAULT ; - addr =gfn_to_hva_memslot (memslot, gfn); -if (kvm_is_error_hva (addr)) -return -EFAULT ; - r =__copy_to_user ((void __user *)addr + offset, data, len); -if (r) -return -EFAULT ; -mark_page_dirty_in_slot (kvm, memslot, gfn); +if (IS_ENABLED (CONFIG_KVM_GMEM_MAPPABLE ) && +kvm_slot_can_be_private (slot) && + !slot->userspace_addr) { +int r =__kvm_write_guest_memfd_page (kvm, slot, gfn, data, + offset, len); + +if (r) +return r; + }else { +unsigned long addr =gfn_to_hva_memslot (slot, gfn); + +if (kvm_is_error_hva (addr)) +return -EFAULT ; +if (__copy_to_user ((void __user *)addr + offset, data, len)) +return -EFAULT ; + } + +mark_page_dirty_in_slot (kvm, slot, gfn);return 0 ; }
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 9/15 KVM: guest_memfd: Add KVM capability to check if guest_memfd is host mappable
In reply to Fuad TabbaAdd the KVM capability KVM_CAP_GUEST_MEMFD_MAPPABLE, which is true if mapping guest memory is supported by the host.
--- include/uapi/linux/kvm.h | 1 + virt/kvm/kvm_main.c | 4 ++++ 2 files changed, 5 insertions(+)
include/uapi/linux/kvm.h +1 -0
@@ -933,6 +933,7 @@ struct kvm_enable_cap {#define KVM_CAP_PRE_FAULT_MEMORY 236#define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237#define KVM_CAP_X86_GUEST_MODE 238 +#define KVM_CAP_GUEST_MEMFD_MAPPABLE 239struct kvm_irq_routing_irqchip {__u32 irqchip;
virt/kvm/kvm_main.c +4 -0
@@ -4979,6 +4979,10 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)#ifdef CONFIG_KVM_PRIVATE_MEM case KVM_CAP_GUEST_MEMFD :return !kvm ||kvm_arch_has_private_mem (kvm); +#endif +#ifdef CONFIG_KVM_GMEM_MAPPABLE +case KVM_CAP_GUEST_MEMFD_MAPPABLE : +return !kvm ||kvm_arch_has_private_mem (kvm);#endif default :break ;
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 10/15 KVM: guest_memfd: Add a guest_memfd() flag to initialize it as mappable
In reply to Fuad TabbaNot all use cases require guest_memfd() to be mappable by the host when first created. Add a new flag, GUEST_MEMFD_FLAG_INIT_MAPPABLE, which when set on KVM_CREATE_GUEST_MEMFD initializes the memory as mappable by the host. Otherwise, memory is private until shared by the guest with the host.
--- Documentation/virt/kvm/api.rst | 4 ++++ include/uapi/linux/kvm.h | 1 + tools/testing/selftests/kvm/guest_memfd_test.c | 7 +++++-- virt/kvm/guest_memfd.c | 6 +++++- 4 files changed, 15 insertions(+), 3 deletions(-)
Documentation/virt/kvm/api.rst +4 -0
@@ -6383,6 +6383,10 @@ most one mapping per page, i.e. binding multiple memory regions to a single guest_memfd range is not allowed (any number of memory regions can be bound to a single guest_memfd file, but the bound ranges must not overlap). +If the capability KVM_CAP_GUEST_MEMFD_MAPPABLE is supported, then the flags +field supports GUEST_MEMFD_FLAG_INIT_MAPPABLE, which initializes the memory +as mappable by the host. + See KVM_SET_USER_MEMORY_REGION2 for additional details. 4.143 KVM_PRE_FAULT_MEMORY
include/uapi/linux/kvm.h +1 -0
@@ -1566,6 +1566,7 @@ struct kvm_memory_attributes {#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3)#define KVM_CREATE_GUEST_MEMFD _IOWR(KVMIO, 0xd4, struct kvm_create_guest_memfd) +#define GUEST_MEMFD_FLAG_INIT_MAPPABLE (1UL << 0)struct kvm_create_guest_memfd {__u64 size;
tools/testing/selftests/kvm/guest_memfd_test.c +5 -2
@@ -123,7 +123,7 @@ static void test_invalid_punch_hole(int fd, size_t page_size, size_t total_size)static void test_create_guest_memfd_invalid (struct kvm_vm *vm) {size_t page_size =getpagesize (); -uint64_t flag; +uint64_t flag =BIT (0 );size_t size;int fd;
@@ -134,7 +134,10 @@ static void test_create_guest_memfd_invalid(struct kvm_vm *vm) size); } -for (flag =BIT (0 ); flag; flag <<=1 ) { +if (kvm_has_cap (KVM_CAP_GUEST_MEMFD_MAPPABLE )) + flag =GUEST_MEMFD_FLAG_INIT_MAPPABLE <<1 ; + +for (; flag; flag <<=1 ) { fd =__vm_create_guest_memfd (vm, page_size, flag); TEST_ASSERT(fd == -1 && errno == EINVAL, "guest_memfd() with flag '0x%lx' should fail with EINVAL",
virt/kvm/guest_memfd.c +5 -1
@@ -939,7 +939,8 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) goto err_gmem; } -if (IS_ENABLED (CONFIG_KVM_GMEM_MAPPABLE )) { +if (IS_ENABLED (CONFIG_KVM_GMEM_MAPPABLE ) && + (flags &GUEST_MEMFD_FLAG_INIT_MAPPABLE )) { err =gmem_set_mappable (file_inode (file),0 , size >>PAGE_SHIFT );if (err) {fput (file);
@@ -968,6 +969,9 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)u64 flags = args->flags;u64 valid_flags =0 ; +if (IS_ENABLED (CONFIG_KVM_GMEM_MAPPABLE )) + valid_flags |=GUEST_MEMFD_FLAG_INIT_MAPPABLE ; +if (flags & ~valid_flags)return -EINVAL ;
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 11/15 KVM: guest_memfd: selftests: guest_memfd mmap() test when mapping is allowed
In reply to Fuad TabbaExpand the guest_memfd selftests to include testing mapping guest memory if the capability is supported, and that still checks that memory is not mappable if the capability isn't supported.
Also, build the guest_memfd selftest for aarch64.
--- tools/testing/selftests/kvm/Makefile | 1 + .../testing/selftests/kvm/guest_memfd_test.c | 57 +++++++++++++++++-- 2 files changed, 53 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/Makefile +1 -0
@@ -174,6 +174,7 @@ TEST_GEN_PROGS_aarch64 += coalesced_io_test TEST_GEN_PROGS_aarch64 += demand_paging_test TEST_GEN_PROGS_aarch64 += dirty_log_test TEST_GEN_PROGS_aarch64 += dirty_log_perf_test +TEST_GEN_PROGS_aarch64 += guest_memfd_test TEST_GEN_PROGS_aarch64 += guest_print_test TEST_GEN_PROGS_aarch64 += get-reg-list TEST_GEN_PROGS_aarch64 += kvm_create_max_vcpus
tools/testing/selftests/kvm/guest_memfd_test.c +52 -5
87 lines not on this page · raw message
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 12/15 KVM: arm64: Skip VMA checks for slots without userspace address
In reply to Fuad TabbaMemory slots backed by guest memory might be created with no intention of being mapped by the host. These are recognized by not having a userspace address in the memory slot.
VMA checks are neither possible nor necessary for this kind of slot, so skip them.
--- arch/arm64/kvm/mmu.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/arm64/kvm/mmu.c +8 -0
@@ -988,6 +988,10 @@ static void stage2_unmap_memslot(struct kvm *kvm,phys_addr_t size =PAGE_SIZE * memslot->npages;hva_t reg_end = hva + size; +/* Host will not map this private memory without a userspace address. */ +if (kvm_slot_can_be_private (memslot) && !hva) +return ; +/* * A memory region could potentially cover multiple VMAs, and any holes * between them, so iterate over all of them to find out if we should
@@ -2133,6 +2137,10 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, hva = new->userspace_addr; reg_end = hva + (new->npages <<PAGE_SHIFT ); +/* Host will not map this private memory without a userspace address. */ +if ((kvm_slot_can_be_private (new)) && !hva) +return 0 ; +mmap_read_lock (current->mm);/* * A memory region could potentially cover multiple VMAs, and any holes
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 13/15 KVM: arm64: Refactor user_mem_abort() calculation of force_pte
In reply to Fuad TabbaTo simplify the code and to make the assumptions clearer, refactor user_mem_abort() by immediately setting force_pte to true if logging_active is true. Also, add a check to ensure that the assumption that logging_active is guaranteed to never be true for VM_PFNMAP memslot is true.
--- arch/arm64/kvm/mmu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
arch/arm64/kvm/mmu.c +7 -5
@@ -1440,7 +1440,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, bool fault_is_perm) {int ret =0 ; -bool write_fault, writable, force_pte = false; +bool write_fault, writable;bool exec_fault, mte_allowed;bool device = false, vfio_allow_any_uc = false;unsigned long mmu_seq;
@@ -1452,6 +1452,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,gfn_t gfn;kvm_pfn_t pfn;bool logging_active =memslot_is_logging (memslot); +bool force_pte = logging_active;long vma_pagesize, fault_granule;enum kvm_pgtable_prot prot =KVM_PGTABLE_PROT_R ;struct kvm_pgtable *pgt;
@@ -1497,12 +1498,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,* logging_active is guaranteed to never be true for VM_PFNMAP * memslots. */ -if (logging_active) { - force_pte = true; +if (WARN_ON_ONCE (logging_active && (vma->vm_flags &VM_PFNMAP ))) +return -EFAULT ; + +if (force_pte) vma_shift =PAGE_SHIFT ; - }else { +else vma_shift =get_vma_page_shift (vma, hva); - }switch (vma_shift) {#ifndef __PAGETABLE_PMD_FOLDED
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 14/15 KVM: arm64: Handle guest_memfd()-backed guest page faults
In reply to Fuad TabbaAdd arm64 support for resolving guest page faults on guest_memfd() backed memslots. This support is not contingent on pKVM, or other confidential computing support, and works in both VHE and nVHE modes.
Without confidential computing, this support is useful for testing and debugging. In the future, it might also be useful should a user want to use guest_memfd() for all code, whether it's for a protected guest or not.
For now, the fault granule is restricted to PAGE_SIZE.
--- arch/arm64/kvm/mmu.c | 86 ++++++++++++++++++++++++++++------------ include/linux/kvm_host.h | 5 +++ virt/kvm/kvm_main.c | 5 --- 3 files changed, 66 insertions(+), 30 deletions(-)
arch/arm64/kvm/mmu.c +61 -25
@@ -1434,6 +1434,39 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma)return vma->vm_flags &VM_MTE_ALLOWED ; } +static kvm_pfn_t faultin_pfn (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,bool write_fault,bool *writable, +struct page **page,bool is_private) +{ +kvm_pfn_t pfn; +int ret; + +if (!is_private) +return __kvm_faultin_pfn (slot, gfn, write_fault ?FOLL_WRITE :0 , writable, page); + + *writable = false; + +if (WARN_ON_ONCE (write_fault &&memslot_is_readonly (slot))) +return KVM_PFN_ERR_NOSLOT_MASK ; + + ret =kvm_gmem_get_pfn (kvm, slot, gfn, &pfn, page,NULL ); +if (!ret) { + *writable = write_fault; +return pfn; + } + +if (ret == -EHWPOISON ) +return KVM_PFN_ERR_HWPOISON ; + +return KVM_PFN_ERR_NOSLOT_MASK ; +} + +static bool is_private_mem (struct kvm *kvm,struct kvm_memory_slot *memslot,phys_addr_t ipa) +{ +return kvm_arch_has_private_mem (kvm) &&kvm_slot_can_be_private (memslot) && + (kvm_mem_is_private (kvm, ipa >>PAGE_SHIFT ) || !memslot->userspace_addr); +} + static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, struct kvm_s2_trans *nested, struct kvm_memory_slot *memslot, unsigned long hva,
@@ -1441,24 +1474,25 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, {int ret =0 ;bool write_fault, writable; -bool exec_fault, mte_allowed; +bool exec_fault, mte_allowed = false;bool device = false, vfio_allow_any_uc = false;unsigned long mmu_seq;phys_addr_t ipa = fault_ipa;struct kvm *kvm = vcpu->kvm;struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; -struct vm_area_struct *vma; +struct vm_area_struct *vma =NULL ;short vma_shift;gfn_t gfn;kvm_pfn_t pfn;bool logging_active =memslot_is_logging (memslot); -bool force_pte = logging_active; -long vma_pagesize, fault_granule; +bool is_private =is_private_mem (kvm, memslot, fault_ipa); +bool force_pte = logging_active || is_private; +long vma_pagesize, fault_granule =PAGE_SIZE ;enum kvm_pgtable_prot prot =KVM_PGTABLE_PROT_R ;struct kvm_pgtable *pgt;struct page *page; -if (fault_is_perm) +if (fault_is_perm && !is_private) fault_granule =kvm_vcpu_trap_get_perm_fault_granule (vcpu); write_fault =kvm_is_write_fault (vcpu); exec_fault =kvm_vcpu_trap_is_exec_fault (vcpu);
@@ -1482,24 +1516,30 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,return ret; } +mmap_read_lock (current->mm); +/* * Let's check if we will get back a huge page backed by hugetlbfs, or * get block mapping for device MMIO region. */ -mmap_read_lock (current->mm); - vma =vma_lookup (current->mm, hva); -if (unlikely (!vma)) { -kvm_err ("Failed to find VMA for hva 0x%lx\n" , hva); -mmap_read_unlock (current->mm); -return -EFAULT ; - } +if (!is_private) { + vma =vma_lookup (current->mm, hva); +if (unlikely (!vma)) { +kvm_err ("Failed to find VMA for hva 0x%lx\n" , hva); +mmap_read_unlock (current->mm); +return -EFAULT ; + } -/* -* logging_active is guaranteed to never be true for VM_PFNMAP -* memslots. -*/ -if (WARN_ON_ONCE (logging_active && (vma->vm_flags &VM_PFNMAP ))) -return -EFAULT ; +/* +* logging_active is guaranteed to never be true for VM_PFNMAP +* memslots. +*/ +if (WARN_ON_ONCE (logging_active && (vma->vm_flags &VM_PFNMAP ))) +return -EFAULT ; + + vfio_allow_any_uc = vma->vm_flags &VM_ALLOW_ANY_UNCACHED ; + mte_allowed =kvm_vma_mte_allowed (vma); + }if (force_pte) vma_shift =PAGE_SHIFT ;
@@ -1570,17 +1610,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, } gfn = ipa >>PAGE_SHIFT ; - mte_allowed =kvm_vma_mte_allowed (vma); - - vfio_allow_any_uc = vma->vm_flags &VM_ALLOW_ANY_UNCACHED ;/* Don't use the VMA after the unlock -- it may have vanished */ vma =NULL ;/* * Read mmu_invalidate_seq so that KVM can detect if the results of -* vma_lookup() or __kvm_faultin_pfn() become stale prior to -* acquiring kvm->mmu_lock. +* vma_lookup() or faultin_pfn() become stale prior to acquiring +* kvm->mmu_lock. * * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs * with the smp_wmb() in kvm_mmu_invalidate_end().
@@ -1588,8 +1625,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, mmu_seq = vcpu->kvm->mmu_invalidate_seq;mmap_read_unlock (current->mm); - pfn =__kvm_faultin_pfn (memslot, gfn, write_fault ?FOLL_WRITE :0 , - &writable, &page); + pfn =faultin_pfn (kvm, memslot, gfn, write_fault, &writable, &page, is_private);if (pfn ==KVM_PFN_ERR_HWPOISON ) {kvm_send_hwpoison_signal (hva, vma_shift);return 0 ;
include/linux/kvm_host.h +5 -0
@@ -1853,6 +1853,11 @@ static inline int memslot_id(struct kvm *kvm, gfn_t gfn)return gfn_to_memslot (kvm, gfn)->id; } +static inline bool memslot_is_readonly (const struct kvm_memory_slot *slot) +{ +return slot->flags &KVM_MEM_READONLY ; +} +static inline gfn_t hva_to_gfn_memslot (unsigned long hva,struct kvm_memory_slot *slot) {
virt/kvm/kvm_main.c +0 -5
@@ -2622,11 +2622,6 @@ unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn)return size; } -static bool memslot_is_readonly (const struct kvm_memory_slot *slot) -{ -return slot->flags &KVM_MEM_READONLY ; -} -static unsigned long __gfn_to_hva_many (const struct kvm_memory_slot *slot,gfn_t gfn,gfn_t *nr_pages,bool write) {
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog
RFC PATCH v5 15/15 KVM: arm64: Enable guest_memfd private memory when pKVM is enabled
In reply to Fuad TabbaImplement kvm_arch_has_private_mem() in arm64 when pKVM is enabled, and make it dependent on the configuration option.
Also, now that the infrastructure is in place for arm64 to support guest private memory, enable it in the arm64 kernel configuration.
--- arch/arm64/include/asm/kvm_host.h | 3 +++ arch/arm64/kvm/Kconfig | 1 + 2 files changed, 4 insertions(+)
arch/arm64/include/asm/kvm_host.h +3 -0
@@ -1529,4 +1529,7 @@ void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);#define kvm_has_s1poe (k) \ (kvm_has_feat((k), ID_AA64MMFR3_EL1, S1POE, IMP)) +#define kvm_arch_has_private_mem (kvm) \ + (IS_ENABLED(CONFIG_KVM_PRIVATE_MEM) && is_protected_kvm_enabled()) +#endif /* __ARM64_KVM_HOST_H__ */
arch/arm64/kvm/Kconfig +1 -0
@@ -38,6 +38,7 @@ menuconfig KVM select HAVE_KVM_VCPU_RUN_PID_CHANGE select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS + select KVM_GMEM_MAPPABLE help Support hosting virtualized guest machines.
Signature · 1 line
2.48.0.rc2.279.g1de40edade-goog