mm/kvm: Improve parallelism for access bit harvesting
Subsystems (13)
v3 cover mm/kvm: Improve parallelism for access bit harvesting
This patchset adds a fast path in KVM to test and clear access bits on sptes without taking the mmu_lock. It also adds support for using a bitmap to (1) test the access bits for many sptes in a single call to mmu_notifier_test_young, and to (2) clear the access bits for many ptes in a single call to mmu_notifier_clear_young.
With Yu's permission, I'm now working on getting this series into a mergeable state.
I'm posting this as an RFC because I'm not sure if the arm64 bits are correct, and I haven't done complete performance testing. I want to do broader experimentation to see how much this improves VM performance in a cloud environment, but I want to be sure that the code is mergeable first.
Yu has posted other performance results[1], [2]. This v3 shouldn't significantly change the x86 results, but the arm64 results may have changed.
The most important changes since v2[3]:
- Split the test_clear_young MMU notifier back into test_young and clear_young. I did this because the bitmap passed in has a distinct meaning for each of them, and I felt that this was cleaner.
- The return value of test_young / clear_young now indicates if the bitmap was used.
- Removed the custom spte walker to implement the lockless path. This was important for arm64 to be functionally correct (thanks Oliver), and it avoids a lot of problems brought up in review of v2 (for example[4]).
- Add kvm_arch_prepare_bitmap_age and kvm_arch_finish_bitmap_age to allow for arm64 to implement its bitmap-based aging to grab the MMU lock for reading while allowing x86 to be lockless.
- The powerpc changes have been dropped.
- The logic to inform architectures how to use the bitmap has been cleaned up (kvm_should_clear_young has been split into kvm_gfn_should_age and kvm_gfn_record_young) (thanks Nicolas).
There were some smaller changes too:
- Added test_clear_young_metadata (thanks Sean).
- MMU_NOTIFIER_RANGE_LOCKLESS has been renamed to MMU_NOTIFIER_YOUNG_FAST, to indicate to the caller that passing a bitmap for MGLRU look-around is likely to be beneficial.
- Cleaned up comments that describe the changes to mmu_notifier_test_young / mmu_notifier_clear_young (thanks Nicolas).
[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/[email protected]/
[3]: https://lore.kernel.org/kvmarm/[email protected]/
[4]: https://lore.kernel.org/all/[email protected]/
James Houghton (5):
mm: Add a bitmap into mmu_notifier_{clear,test}_young
KVM: Move MMU notifier function declarations
KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young
KVM: x86: Participate in bitmap-based PTE aging
KVM: arm64: Participate in bitmap-based PTE aging
Yu Zhao (2):
KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask
mm: multi-gen LRU: use mmu_notifier_test_clear_young()
Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/include/asm/kvm_host.h | 5 + arch/arm64/include/asm/kvm_pgtable.h | 4 +- arch/arm64/kvm/hyp/pgtable.c | 21 +- arch/arm64/kvm/mmu.c | 23 ++- arch/x86/include/asm/kvm_host.h | 20 ++ arch/x86/kvm/mmu.h | 6 - arch/x86/kvm/mmu/mmu.c | 16 +- arch/x86/kvm/mmu/spte.h | 1 - arch/x86/kvm/mmu/tdp_mmu.c | 10 +- include/linux/kvm_host.h | 101 ++++++++-- include/linux/mmu_notifier.h | 93 ++++++++- include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 13 +- mm/mmu_notifier.c | 20 +- mm/rmap.c | 9 +- mm/vmscan.c | 183 ++++++++++++++---- virt/kvm/kvm_main.c | 100 +++++++--- 18 files changed, 509 insertions(+), 128 deletions(-)
base-commit: 0cef2c0a2a356137b170c3cb46cb9c1dd2ca3e6b
Signature · 1 line
2.44.0.478.gd926399ef9-goog
PATCH v3 1/7 mm: Add a bitmap into mmu_notifier_{clear,test}_young
The bitmap is provided for secondary MMUs to use if they support it. For test_young(), after it returns, the bitmap represents the pages that were young in the interval [start, end). For clear_young, it represents the pages that we wish the secondary MMU to clear the accessed/young bit for.
If a bitmap is not provided, the mmu_notifier_{test,clear}_young() API should be unchanged except that if young PTEs are found and the architecture supports passing in a bitmap, instead of returning 1, MMU_NOTIFIER_YOUNG_FAST is returned.
This allows MGLRU's look-around logic to work faster, resulting in a 4% improvement in real workloads[1]. Also introduce MMU_NOTIFIER_YOUNG_FAST to indicate to main mm that doing look-around is likely to be beneficial.
If the secondary MMU doesn't support the bitmap, it must return an int that contains MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.
[1]: https://lore.kernel.org/all/[email protected]/
--- include/linux/mmu_notifier.h | 93 +++++++++++++++++++++++++++++++++--- include/trace/events/kvm.h | 13 +++-- mm/mmu_notifier.c | 20 +++++--- virt/kvm/kvm_main.c | 19 ++++++-- 4 files changed, 123 insertions(+), 22 deletions(-)
include/linux/mmu_notifier.h +86 -7
@@ -61,6 +61,10 @@ enum mmu_notifier_event {#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1) +#define MMU_NOTIFIER_YOUNG_FAST (1 << 2) +struct mmu_notifier_ops {/* * Called either by mmu_notifier_unregister or when the mm is
@@ -106,21 +110,36 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*clear_young)(struct mmu_notifier *subscription,struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +unsigned long *bitmap);/* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*test_young)(struct mmu_notifier *subscription,struct mm_struct *mm, -unsigned long address); +unsigned long start, +unsigned long end, +unsigned long *bitmap);/* * change_pte is called in cases that pte mapping to page is changed:
@@ -388,10 +407,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm,unsigned long start, unsigned long end);extern int __mmu_notifier_clear_young (struct mm_struct *mm, -unsigned long start, -unsigned long end); +unsigned long start,unsigned long end, +unsigned long *bitmap);extern int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address); +unsigned long start,unsigned long end, +unsigned long *bitmap);extern void __mmu_notifier_change_pte (struct mm_struct *mm,unsigned long address,pte_t pte);extern int __mmu_notifier_invalidate_range_start (struct mmu_notifier_range *r);
@@ -427,7 +447,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) {if (mm_has_notifiers (mm)) -return __mmu_notifier_clear_young (mm, start, end); +return __mmu_notifier_clear_young (mm, start, end,NULL ); +return 0 ; +} + +/* +* When @bitmap is not provided, clear the young bits in the secondary +* MMUs for all of the pages in the interval [start, end). +* +* If any subscribed secondary MMU does not support @bitmap, this function +* will return an integer containing MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* Some work may have been done in the secondary MMU. +*/ +static inline int mmu_notifier_clear_young_bitmap (struct mm_struct *mm, +unsigned long start, +unsigned long end, +unsigned long *bitmap) +{ +if (mm_has_notifiers (mm)) +return __mmu_notifier_clear_young (mm, start, end, bitmap);return 0 ; }
@@ -435,7 +473,25 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) {if (mm_has_notifiers (mm)) -return __mmu_notifier_test_young (mm, address); +return __mmu_notifier_test_young (mm, address, address +1 , +NULL ); +return 0 ; +} + +/* +* When @bitmap is not provided, test the young bits in the secondary +* MMUs for all of the pages in the interval [start, end). +* +* If any subscribed secondary MMU does not support @bitmap, this function +* will return an integer containing MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +*/ +static inline int mmu_notifier_test_young_bitmap (struct mm_struct *mm, +unsigned long start, +unsigned long end, +unsigned long *bitmap) +{ +if (mm_has_notifiers (mm)) +return __mmu_notifier_test_young (mm, start, end, bitmap);return 0 ; }
@@ -644,12 +700,35 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_clear_young (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +return 0 ; +} + +static inline int mmu_notifier_clear_young_bitmap (struct mm_struct *mm, +unsigned long start, +unsigned long end, +unsigned long *bitmap) +{ +return 0 ; +} +static inline int mmu_notifier_test_young (struct mm_struct *mm,unsigned long address) {return 0 ; } +static inline int mmu_notifier_test_young_bitmap (struct mm_struct *mm, +unsigned long start, +unsigned long end, +unsigned long *bitmap) +{ +return 0 ; +} +static inline void mmu_notifier_change_pte (struct mm_struct *mm,unsigned long address,pte_t pte) {
include/trace/events/kvm.h +8 -5
@@ -490,18 +490,21 @@ TRACE_EVENT(kvm_age_hva, );TRACE_EVENT (kvm_test_age_hva, - TP_PROTO(unsigned long hva), -TP_ARGS (hva), + TP_PROTO(unsigned long start, unsigned long end), +TP_ARGS (start, end),TP_STRUCT__entry ( - __field( unsigned long, hva ) + __field( unsigned long, start ) + __field( unsigned long, end ) ),TP_fast_assign ( - __entry->hva = hva; + __entry->start = start; + __entry->end = end; ), -TP_printk ("mmu notifier test age hva: %#016lx" , __entry->hva) +TP_printk ("mmu notifier test age hva: %#016lx -- %#016lx" , + __entry->start, __entry->end) );#endif /* _TRACE_KVM_MAIN_H */
mm/mmu_notifier.c +14 -6
@@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +unsigned long *bitmap) {struct mmu_notifier *subscription;int young =0 , id;
@@ -395,7 +396,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, srcu_read_lock_held(&srcu)) {if (subscription->ops->clear_young) young |= subscription->ops->clear_young (subscription, - mm, start, end); + mm, start, end, + bitmap); }srcu_read_unlock (&srcu, id);
@@ -403,7 +405,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, }int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address) +unsigned long start,unsigned long end, +unsigned long *bitmap) {struct mmu_notifier *subscription;int young =0 , id;
@@ -413,9 +416,14 @@ int __mmu_notifier_test_young(struct mm_struct *mm, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) {if (subscription->ops->test_young) { - young = subscription->ops->test_young (subscription, mm, - address); -if (young) + young |= subscription->ops->test_young (subscription, mm, + start, end, + bitmap); +if (young && !bitmap) +/* +* We're not using a bitmap, so there is no +* need to check any more secondary MMUs. +*/ break ; } }
virt/kvm/kvm_main.c +15 -4
@@ -917,10 +917,15 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,static int kvm_mmu_notifier_clear_young (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +unsigned long *bitmap) {trace_kvm_age_hva (start, end); +/* We don't support bitmaps. Don't test or clear anything. */ +if (bitmap) +return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ; +/* * Even though we do not flush TLB, this will still adversely * affect performance on pre-Haswell Intel EPT, where there is
@@ -939,11 +944,17 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,static int kvm_mmu_notifier_test_young (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long address) +unsigned long start, +unsigned long end, +unsigned long *bitmap) { -trace_kvm_test_age_hva (address); +trace_kvm_test_age_hva (start, end); + +/* We don't support bitmaps. Don't test or clear anything. */ +if (bitmap) +return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ; -return kvm_handle_hva_range_no_flush (mn, address, address +1 , +return kvm_handle_hva_range_no_flush (mn, start, end, kvm_test_age_gfn); }
Signature · 1 line
2.44.0.478.gd926399ef9-goog
Re: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young
Quoted 35 lines · James Houghton
The bitmap is provided for secondary MMUs to use if they support it. For
test_young(), after it returns, the bitmap represents the pages that were young in the interval [start, end). Forclear_young, it represents the pages that we wish the secondary MMU to clear the accessed/young bit for.If a bitmap is not provided, the mmu_notifier_{test,clear}_young() API should be unchanged except that if young PTEs are found and the architecture supports passing in a bitmap, instead of returning 1,
MMU_NOTIFIER_YOUNG_FASTis returned.This allows MGLRU's look-around logic to work faster, resulting in a 4% improvement in real workloads[1]. Also introduce
MMU_NOTIFIER_YOUNG_FASTto indicate to main mm that doing look-around is likely to be beneficial.If the secondary MMU doesn't support the bitmap, it must return an int that contains
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.[1]: https://lore.kernel.org/all/[email protected]/
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- include/linux/mmu_notifier.h | 93 +++++++++++++++++++++++++++++++++--- include/trace/events/kvm.h | 13 +++-- mm/mmu_notifier.c | 20 +++++--- virt/kvm/kvm_main.c | 19 ++++++-- 4 files changed, 123 insertions(+), 22 deletions(-)include/linux/mmu_notifier.h +0
-0
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 65
Quoted 5 lines
include/linux/mmu_notifier.h +2
-0@@ -61,6 +61,10 @@ enum mmu_notifier_event {#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1)
Especially this one really deserves some documentation :)
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 66
Quoted 1 line
include/linux/mmu_notifier.h +1
-0+#define MMU_NOTIFIER_YOUNG_FAST (1 << 2)
And that one as well.
Likely best to briefly document all of them, and how they are supposed to be used (return value for X).
Quoted 30 lines
include/linux/mmu_notifier.h +12
-1+struct mmu_notifier_ops {/* * Called either by mmu_notifier_unregister or when the mm is @@ -106,21 +110,36 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*clear_young)(struct mmu_notifier *subscription,struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +unsigned long *bitmap);/* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 142
Quoted 10 lines
include/linux/mmu_notifier.h +6
-1+* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*test_young)(struct mmu_notifier *subscription,struct mm_struct *mm, -unsigned long address); +unsigned long start, +unsigned long end, +unsigned long *bitmap);
What does "quickly" mean (why not use "fast")? What are the semantics, I don't find any existing usage of that in this file.
Further, what is MMU_NOTIFIER_YOUNG you introduce used for?
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young
Ah, I didn't see this in my inbox, sorry David!
Quoted 8 lines · David Hildenbrand
Quoted 6 lines · James Houghton
include/linux/mmu_notifier.h +0
-0@@ -61,6 +61,10 @@ enum mmu_notifier_event {
Quoted 6 lines
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 65
Quoted 4 lines
include/linux/mmu_notifier.h +2
-0#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1)Especially this one really deserves some documentation :)
Yes, will do. Something like
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE indicates that the passed-in
bitmap either (1) does not accurately represent the age of the pages (in the case of test_young), or (2) was not able to be used to completely clear the age/access bit (in the case of clear_young).
Quoted 4 lines
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 66
Quoted 2 lines
include/linux/mmu_notifier.h +1
-0+#define MMU_NOTIFIER_YOUNG_FAST (1 << 2)And that one as well.
Something like
Indicates that (1) passing a bitmap ({test,clear}_young_bitmap)
would have been supported for this address range.
The name MMU_NOTIFIER_YOUNG_FAST really comes from the fact that KVM is able to harvest the access bit "fast" (so for x86, locklessly, and for arm64, with the KVM MMU read lock), "fast" enough that using a bitmap to do look-around is probably a good idea.
Quoted 3 lines
Likely best to briefly document all of them, and how they are supposed to be used (return value for X).
Right. Will do.
Quoted 36 lines
Quoted 36 lines
include/linux/mmu_notifier.h +15
-1+struct mmu_notifier_ops {/* * Called either by mmu_notifier_unregister or when the mm is @@ -106,21 +110,36 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*clear_young)(struct mmu_notifier *subscription,struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +unsigned long *bitmap);/* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*test_young)(struct mmu_notifier *subscription,
Quoted 8 lines
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 142
Quoted 5 lines
include/linux/mmu_notifier.h +3
-1struct mm_struct *mm, - unsigned long address); + unsigned long start, + unsigned long end, + unsigned long *bitmap);What does "quickly" mean (why not use "fast")? What are the semantics, I don't find any existing usage of that in this file.
"fast" means fast enough such that using a bitmap to scan adjacent pages (e.g. with MGLRU) is likely to be beneficial. I'll write more in this comment. Perhaps I should just rename it to MMU_NOTIFIER_YOUNG_BITMAP_SUPPORTED and drop the whole "likely to be beneficial" thing -- that's for MGLRU/etc. to decide really.
Quoted 2 lines
Further, what is
MMU_NOTIFIER_YOUNGyou introduce used for?
MMU_NOTIFIER_YOUNG is the return value when the page was young, but we
(1) didn't use a bitmap, and (2) the "fast" access bit harvesting wasn't possible. In this case we simply return 1, which is MMU_NOTIFIER_YOUNG. I'll make kvm_mmu_notifier_test_clear_young() properly return MMU_NOTIFIER_YOUNG instead of relying on the fact that it will be 1.
Thanks David!
Re: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young
Quoted 1 line · James Houghton
Ah, I didn't see this in my inbox, sorry David!
No worries :)
Quoted 17 lines · David Hildenbrand
Quoted 8 lines · James Houghton
Quoted 6 lines
include/linux/mmu_notifier.h +0
-0@@ -61,6 +61,10 @@ enum mmu_notifier_event {Quoted 6 lines
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 65
Quoted 4 lines
include/linux/mmu_notifier.h +2
-0#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1)Especially this one really deserves some documentation :)
Quoted 6 lines
Yes, will do. Something like
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEindicates that the passed-in
bitmap either (1) does not accurately represent the age of the pages (in the case oftest_young), or (2) was not able to be used to completely clear the age/access bit (in the case ofclear_young).
Make sense. I do wonder what the expected reaction from the caller is :)
Quoted 8 lines
Quoted 5 lines
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 66
Quoted 3 lines
include/linux/mmu_notifier.h +1
-0+#define MMU_NOTIFIER_YOUNG_FAST (1 << 2)And that one as well.
Something like
Quoted 7 lines
Indicates that (1) passing a bitmap ({test,clear}_young_bitmap)
would have been supported for this address range.The name
MMU_NOTIFIER_YOUNG_FASTreally comes from the fact that KVM is able to harvest the access bit "fast" (so for x86, locklessly, and for arm64, with the KVM MMU read lock), "fast" enough that using a bitmap to do look-around is probably a good idea.
Is that really the right way to communicate that ("would have been supported") -- wouldn't we want to sense support differently?
Quoted 52 lines
Quoted 4 lines
Likely best to briefly document all of them, and how they are supposed to be used (return value for X).
Right. Will do.
Quoted 36 lines
Quoted 36 lines
include/linux/mmu_notifier.h +15
-1+struct mmu_notifier_ops {/* * Called either by mmu_notifier_unregister or when the mm is @@ -106,21 +110,36 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*clear_young)(struct mmu_notifier *subscription,struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +unsigned long *bitmap);/* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* If @bitmap is given but is not supported, return +* MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. +* +* If the walk is done "quickly" and there were young PTEs, +* MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*test_young)(struct mmu_notifier *subscription,Quoted 8 lines
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 142
Quoted 5 lines
include/linux/mmu_notifier.h +3
-1struct mm_struct *mm, - unsigned long address); + unsigned long start, + unsigned long end, + unsigned long *bitmap);What does "quickly" mean (why not use "fast")? What are the semantics, I don't find any existing usage of that in this file.
Quoted 5 lines
"fast" means fast enough such that using a bitmap to scan adjacent pages (e.g. with MGLRU) is likely to be beneficial. I'll write more in this comment. Perhaps I should just rename it to
MMU_NOTIFIER_YOUNG_BITMAP_SUPPORTEDand drop the whole "likely to be beneficial" thing -- that's for MGLRU/etc. to decide really.
Yes!
Quoted 10 lines
Quoted 3 lines
Further, what is
MMU_NOTIFIER_YOUNGyou introduce used for?
MMU_NOTIFIER_YOUNGis the return value when the page was young, but we
(1) didn't use a bitmap, and (2) the "fast" access bit harvesting wasn't possible. In this case we simply return 1, which isMMU_NOTIFIER_YOUNG. I'll makekvm_mmu_notifier_test_clear_young()properly returnMMU_NOTIFIER_YOUNGinstead of relying on the fact that it will be 1.
Yes, that will clarify it!
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young
Quoted 23 lines · David Hildenbrand
Quoted 1 line · James Houghton
Ah, I didn't see this in my inbox, sorry David!
No worries :)
Quoted 17 lines · David Hildenbrand
Quoted 14 lines · James Houghton
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 65
Quoted 10 lines
include/linux/mmu_notifier.h +2
-0@@ -61,6 +61,10 @@ enum mmu_notifier_event {#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1)Especially this one really deserves some documentation :)
Quoted 8 lines
Quoted 6 lines
Yes, will do. Something like
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEindicates that the passed-in
bitmap either (1) does not accurately represent the age of the pages (in the case oftest_young), or (2) was not able to be used to completely clear the age/access bit (in the case ofclear_young).Make sense. I do wonder what the expected reaction from the caller is :)
In this series the caller doesn't actually care (matching what Yu had in his v2[1]). test_spte_young() probably ought to return false if it finds MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (and I'll do this in v4 if no one objects), but it doesn't have to. The bitmap will never say that a page is young when it was actually not, only the other way around.
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 66
Quoted 4 lines
include/linux/mmu_notifier.h +1
-0+#define MMU_NOTIFIER_YOUNG_FAST (1 << 2)And that one as well.
Something like
Indicates that (1) passing a bitmap ({test,clear}_young_bitmap)
would have been supported for this address range.
Quoted 7 lines
Quoted 4 lines
The name
MMU_NOTIFIER_YOUNG_FASTreally comes from the fact that KVM is able to harvest the access bit "fast" (so for x86, locklessly, and for arm64, with the KVM MMU read lock), "fast" enough that using a bitmap to do look-around is probably a good idea.Is that really the right way to communicate that ("would have been supported") -- wouldn't we want to sense support differently?
What I have now seems fine to me. It would be a little nicer to have a way to query for bitmap support and make sure that the answer will not be stale by the time we call the bitmap notifiers, but the complexity to make that work seems unnecessary for dealing with such an uncommon scenario.
Maybe the right thing to do is just to have KVM always return the same answer. So instead of checking if the shadow root is allocated, we could check something else (I'm not sure what exactly yet though...).
Re: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young
In reply to James HoughtonQuoted 35 lines · James Houghton
The bitmap is provided for secondary MMUs to use if they support it. For
test_young(), after it returns, the bitmap represents the pages that were young in the interval [start, end). Forclear_young, it represents the pages that we wish the secondary MMU to clear the accessed/young bit for.If a bitmap is not provided, the mmu_notifier_{test,clear}_young() API should be unchanged except that if young PTEs are found and the architecture supports passing in a bitmap, instead of returning 1,
MMU_NOTIFIER_YOUNG_FASTis returned.This allows MGLRU's look-around logic to work faster, resulting in a 4% improvement in real workloads[1]. Also introduce
MMU_NOTIFIER_YOUNG_FASTto indicate to main mm that doing look-around is likely to be beneficial.If the secondary MMU doesn't support the bitmap, it must return an int that contains
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.[1]: https://lore.kernel.org/all/[email protected]/
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- include/linux/mmu_notifier.h | 93 +++++++++++++++++++++++++++++++++--- include/trace/events/kvm.h | 13 +++-- mm/mmu_notifier.c | 20 +++++--- virt/kvm/kvm_main.c | 19 ++++++-- 4 files changed, 123 insertions(+), 22 deletions(-)include/linux/mmu_notifier.h +0
-0
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 65
Quoted 5 lines
include/linux/mmu_notifier.h +2
-0@@ -61,6 +61,10 @@ enum mmu_notifier_event {#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1)
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE appears to be unused by all callers of test/clear_young(). I would vote to remove it.
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 66
Quoted 1 line
include/linux/mmu_notifier.h +1
-0+#define MMU_NOTIFIER_YOUNG_FAST (1 << 2)
Instead of MMU_NOTIFIER_YOUNG_FAST, how about MMU_NOTIFIER_YOUNG_LOOK_AROUND? i.e. The secondary MMU is returning saying it recommends doing a look-around and passing in a bitmap?
That would avoid the whole "what does FAST really mean" confusion.
Quoted 7 lines
virt/kvm/kvm_main.c +0
-0@@ -917,10 +917,15 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, struct mm_struct *mm,
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, · line 927
Quoted 10 lines
virt/kvm/kvm_main.c +5
-1unsigned long start, - unsigned long end) + unsigned long end, + unsigned long *bitmap) {trace_kvm_age_hva (start ,end ); +/* We don't support bitmaps. Don't test or clear anything. */ +if (bitmap) +return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ;
Wouldn't it be a bug to get a bitmap here? The main MM is only suppost to pass in a bitmap if the secondary MMU returns MMU_NOTIFIER_YOUNG_FAST, which KVM does not do at this point.
Put another way, this check seems unneccessary.
Quoted 9 lines
virt/kvm/kvm_main.c +1
-1+/* * Even though we do not flush TLB, this will still adversely * affect performance on pre-Haswell Intel EPT, where there is @@ -939,11 +944,17 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, struct mm_struct *mm, -unsigned long address)
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, · line 955
Quoted 10 lines
virt/kvm/kvm_main.c +8
-1+unsigned long start, + unsigned long end, + unsigned long *bitmap) { -trace_kvm_test_age_hva (address); +trace_kvm_test_age_hva (start ,end ); + +/* We don't support bitmaps. Don't test or clear anything. */ +if (bitmap) +return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ;
Same thing here.
Re: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young
Quoted 38 lines · David Matlack
Quoted 36 lines · James Houghton
The bitmap is provided for secondary MMUs to use if they support it. For
test_young(), after it returns, the bitmap represents the pages that were young in the interval [start, end). Forclear_young, it represents the pages that we wish the secondary MMU to clear the accessed/young bit for.If a bitmap is not provided, the mmu_notifier_{test,clear}_young() API should be unchanged except that if young PTEs are found and the architecture supports passing in a bitmap, instead of returning 1,
MMU_NOTIFIER_YOUNG_FASTis returned.This allows MGLRU's look-around logic to work faster, resulting in a 4% improvement in real workloads[1]. Also introduce
MMU_NOTIFIER_YOUNG_FASTto indicate to main mm that doing look-around is likely to be beneficial.If the secondary MMU doesn't support the bitmap, it must return an int that contains
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.[1]: https://lore.kernel.org/all/[email protected]/
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- include/linux/mmu_notifier.h | 93 +++++++++++++++++++++++++++++++++--- include/trace/events/kvm.h | 13 +++-- mm/mmu_notifier.c | 20 +++++--- virt/kvm/kvm_main.c | 19 ++++++-- 4 files changed, 123 insertions(+), 22 deletions(-)include/linux/mmu_notifier.h +0
-0@@ -61,6 +61,10 @@ enum mmu_notifier_event {
Quoted 7 lines
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 65
Quoted 4 lines
include/linux/mmu_notifier.h +2
-0#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1)
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEappears to be unused by all callers of test/clear_young(). I would vote to remove it.
Works for me.
Quoted 8 lines
include/linux/mmu_notifier.h enum mmu_notifier_event { · line 66
Quoted 2 lines
include/linux/mmu_notifier.h +1
-0+#define MMU_NOTIFIER_YOUNG_FAST (1 << 2)Instead of
MMU_NOTIFIER_YOUNG_FAST, how aboutMMU_NOTIFIER_YOUNG_LOOK_AROUND? i.e. The secondary MMU is returning saying it recommends doing a look-around and passing in a bitmap?That would avoid the whole "what does FAST really mean" confusion.
I think MMU_NOTIFIER_YOUNG_LOOK_AROUND is fine.
Quoted 31 lines
Quoted 8 lines
virt/kvm/kvm_main.c +0
-0@@ -917,10 +917,15 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, struct mm_struct *mm,virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, · line 927
Quoted 10 lines
virt/kvm/kvm_main.c +5
-1unsigned long start, - unsigned long end) + unsigned long end, + unsigned long *bitmap) {trace_kvm_age_hva (start ,end ); +/* We don't support bitmaps. Don't test or clear anything. */ +if (bitmap) +return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ;Wouldn't it be a bug to get a bitmap here? The main MM is only suppost to pass in a bitmap if the secondary MMU returns
MMU_NOTIFIER_YOUNG_FAST, which KVM does not do at this point.Put another way, this check seems unneccessary.
Quoted 6 lines
virt/kvm/kvm_main.c +1
-0+/* * Even though we do not flush TLB, this will still adversely * affect performance on pre-Haswell Intel EPT, where there is @@ -939,11 +944,17 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
Quoted 15 lines
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, · line 955
Quoted 13 lines
virt/kvm/kvm_main.c +8
-2static int kvm_mmu_notifier_test_young (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long address) +unsigned long start, +unsigned long end, +unsigned long *bitmap) { -trace_kvm_test_age_hva (address); +trace_kvm_test_age_hva (start, end); + +/* We don't support bitmaps. Don't test or clear anything. */ +if (bitmap) +return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ;Same thing here.
I will remove them, they are indeed unnecessary, as it is just dead code.
PATCH v3 2/7 KVM: Move MMU notifier function declarations
In reply to James HoughtonTo allow new MMU-notifier-related functions to use gfn_to_hva_memslot(), move some declarations around.
Also move mmu_notifier_to_kvm() for wider use later.
--- include/linux/kvm_host.h | 41 +++++++++++++++++++++------------------- virt/kvm/kvm_main.c | 5 ----- 2 files changed, 22 insertions(+), 24 deletions(-)
include/linux/kvm_host.h +22 -19
@@ -257,25 +257,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER -union kvm_mmu_notifier_arg { -pte_t pte; -unsigned long attributes; -}; - -struct kvm_gfn_range { -struct kvm_memory_slot *slot; -gfn_t start; -gfn_t end; -union kvm_mmu_notifier_arg arg; -bool may_block; -}; -bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); -bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range); -bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range); -bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range); -#endif -enum {OUTSIDE_GUEST_MODE ,IN_GUEST_MODE ,
@@ -2012,6 +1993,11 @@ extern const struct kvm_stats_header kvm_vcpu_stats_header;extern const struct _kvm_stats_desc kvm_vcpu_stats_desc[];#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER +static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) +{ + return container_of(mn, struct kvm, mmu_notifier); +} +static inline int mmu_invalidate_retry (struct kvm *kvm,unsigned long mmu_seq) {if (unlikely (kvm->mmu_invalidate_in_progress))
@@ -2089,6 +2075,23 @@ static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, return READ_ONCE(kvm->mmu_invalidate_seq) != mmu_seq; } + +union kvm_mmu_notifier_arg { +pte_t pte; +unsigned long attributes; +}; + +struct kvm_gfn_range { +struct kvm_memory_slot *slot; +gfn_t start; +gfn_t end; +union kvm_mmu_notifier_arg arg; +bool may_block; +}; +bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); +bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range); +bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range); +bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);#endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING
virt/kvm/kvm_main.c +0 -5
@@ -534,11 +534,6 @@ void kvm_destroy_vcpus(struct kvm *kvm)EXPORT_SYMBOL_GPL (kvm_destroy_vcpus);#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER -static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) -{ - return container_of(mn, struct kvm, mmu_notifier); -} -typedef bool (*gfn_handler_t )(struct kvm *kvm,struct kvm_gfn_range *range);typedef void (*on_lock_fn_t )(struct kvm *kvm);
Signature · 1 line
2.44.0.478.gd926399ef9-goog
PATCH v3 3/7 KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young
In reply to James HoughtonAdd kvm_arch_prepare_bitmap_age() for architectures to indiciate that they support bitmap-based aging in kvm_mmu_notifier_test_clear_young() and that they do not need KVM to grab the MMU lock for writing. This function allows architectures to do other locking or other preparatory work that it needs.
If an architecture does not implement kvm_arch_prepare_bitmap_age() or is unable to do bitmap-based aging at runtime (and marks the bitmap as unreliable):
1. If a bitmap was provided, we inform the caller that the bitmap is unreliable (MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE).
2. If a bitmap was not provided, fall back to the old logic.
Also add logic for architectures to easily use the provided bitmap if they are able. The expectation is that the architecture's implementation of kvm_gfn_test_age() will use kvm_gfn_record_young(), and kvm_gfn_age() will use kvm_gfn_should_age().
--- include/linux/kvm_host.h | 60 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 92 +++++++++++++++++++++++++++++----------- 2 files changed, 127 insertions(+), 25 deletions(-)
include/linux/kvm_host.h +60 -0
@@ -1992,6 +1992,26 @@ extern const struct _kvm_stats_desc kvm_vm_stats_desc[];extern const struct kvm_stats_header kvm_vcpu_stats_header;extern const struct _kvm_stats_desc kvm_vcpu_stats_desc[]; +/* +* Architectures that support using bitmaps for kvm_age_gfn() and +* kvm_test_age_gfn should return true for kvm_arch_prepare_bitmap_age() +* and do any work they need to prepare. The subsequent walk will not +* automatically grab the KVM MMU lock, so some architectures may opt +* to grab it. +* +* If true is returned, a subsequent call to kvm_arch_finish_bitmap_age() is +* guaranteed. +*/ +#ifndef kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +return false; +} +#endif +#ifndef kvm_arch_finish_bitmap_age +static inline void kvm_arch_finish_bitmap_age (struct mmu_notifier *mn) {} +#endif +#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {
@@ -2076,9 +2096,16 @@ static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm,return READ_ONCE (kvm->mmu_invalidate_seq) != mmu_seq; } +struct test_clear_young_metadata { +unsigned long *bitmap; +unsigned long bitmap_offset_end; +unsigned long end; +bool unreliable; +};union kvm_mmu_notifier_arg {pte_t pte;unsigned long attributes; +struct test_clear_young_metadata *metadata; };struct kvm_gfn_range {
@@ -2087,11 +2114,44 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range); + +static inline void kvm_age_set_unreliable (struct kvm_gfn_range *range) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + + args->unreliable = true; +} +static inline unsigned long kvm_young_bitmap_offset (struct kvm_gfn_range *range, +gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +return hva_to_gfn_memslot (args->end -1 , range->slot) - gfn; +} +static inline void kvm_gfn_record_young (struct kvm_gfn_range *range,gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +WARN_ON_ONCE (gfn < range->start || gfn >= range->end); +if (args->bitmap) +__set_bit (kvm_young_bitmap_offset (range, gfn), args->bitmap); +} +static inline bool kvm_gfn_should_age (struct kvm_gfn_range *range,gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +WARN_ON_ONCE (gfn < range->start || gfn >= range->end); +if (args->bitmap) +return test_bit (kvm_young_bitmap_offset (range, gfn), + args->bitmap); +return true; +}#endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING
virt/kvm/kvm_main.c +67 -25
154 lines not on this page · raw message
Signature · 1 line
2.44.0.478.gd926399ef9-goog
Re: [PATCH v3 3/7] KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young
Quoted 5 lines · James Houghton
Add
kvm_arch_prepare_bitmap_age()for architectures to indiciate that they support bitmap-based aging inkvm_mmu_notifier_test_clear_young()and that they do not need KVM to grab the MMU lock for writing. This function allows architectures to do other locking or other preparatory work that it needs.
There's a lot going on here. I know it's extra work but I think the series would be easier to understand and simplify if you introduced the KVM support for lockless test/clear_young() first, and then introduce support for the bitmap-based look-around.
Specifically:
1. Make all test/clear_young() notifiers lockless. i.e. Move the mmu_lock into the architecture-specific code (kvm_age_gfn() and kvm_test_age_gfn()).
2. Convert KVM/x86's kvm_{test,}_age_gfn() to be lockless for the TDP MMU.
4. Convert KVM/arm64's kvm_{test,}_age_gfn() to hold the mmu_lock in read-mode.
5. Add bitmap-based look-around support to KVM/x86 and KVM/arm64
(probably 2-3 patches).
Quoted 30 lines
If an architecture does not implement
kvm_arch_prepare_bitmap_age()or is unable to do bitmap-based aging at runtime (and marks the bitmap as unreliable):
1. If a bitmap was provided, we inform the caller that the bitmap is unreliable (MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE).
2. If a bitmap was not provided, fall back to the old logic.Also add logic for architectures to easily use the provided bitmap if they are able. The expectation is that the architecture's implementation of
kvm_gfn_test_age()will usekvm_gfn_record_young(), andkvm_gfn_age()will usekvm_gfn_should_age().Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- include/linux/kvm_host.h | 60 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 92 +++++++++++++++++++++++++++++----------- 2 files changed, 127 insertions(+), 25 deletions(-)include/linux/kvm_host.h +2
-0@@ -1992,6 +1992,26 @@ extern const struct _kvm_stats_desc kvm_vm_stats_desc[];extern const struct kvm_stats_header kvm_vcpu_stats_header;extern const struct _kvm_stats_desc kvm_vcpu_stats_desc[]; +/* +* Architectures that support using bitmaps for kvm_age_gfn() and
include/linux/kvm_host.h extern const struct _kvm_stats_desc kvm_vm_stats_desc[]; · line 2006
Quoted 10 lines
include/linux/kvm_host.h +10
-0+* kvm_test_age_gfn should return true for kvm_arch_prepare_bitmap_age() +* and do any work they need to prepare. The subsequent walk will not +* automatically grab the KVM MMU lock, so some architectures may opt +* to grab it. +* +* If true is returned, a subsequent call to kvm_arch_finish_bitmap_age() is +* guaranteed. +*/ +#ifndef kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn)
I find the name of these architecture callbacks misleading/confusing. The lockless path is used even when a bitmap is not provided. i.e. bitmap can be NULL in between kvm_arch_prepare/finish_bitmap_age().
include/linux/kvm_host.h extern const struct _kvm_stats_desc kvm_vm_stats_desc[]; · line 2013
Quoted 7 lines
include/linux/kvm_host.h +7
-0+{ +return false; +} +#endif +#ifndef kvm_arch_finish_bitmap_age +static inline void kvm_arch_finish_bitmap_age (struct mmu_notifier *mn) {} +#endif
kvm_arch_finish_bitmap_age() seems unnecessary. I think the KVM/arm64 code could acquire/release the mmu_lock in read-mode in kvm_test_age_gfn() and kvm_age_gfn() right?
include/linux/kvm_host.h static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, · line 2101
Quoted 11 lines
include/linux/kvm_host.h +4
-0+#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -2076,9 +2096,16 @@ static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm,return READ_ONCE (kvm->mmu_invalidate_seq) != mmu_seq; } +struct test_clear_young_metadata { +unsigned long *bitmap; +unsigned long bitmap_offset_end;
bitmap_offset_end is unused.
include/linux/kvm_host.h static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, · line 2108
Quoted 7 lines
include/linux/kvm_host.h +4
-0+unsigned long end; +bool unreliable; +};union kvm_mmu_notifier_arg {pte_t pte;unsigned long attributes; +struct test_clear_young_metadata *metadata;
nit: Maybe s/metadata/test_clear_young/ ?
include/linux/kvm_host.h struct kvm_gfn_range { · line 2117
Quoted 8 lines
include/linux/kvm_host.h +1
-0};struct kvm_gfn_range {@@ -2087,11 +2114,44 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless;
Please document this as it's somewhat subtle. A reader might think this implies the entire operation runs without taking the mmu_lock.
Quoted 129 lines
include/linux/kvm_host.h +32
-0};bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range); + +static inline void kvm_age_set_unreliable (struct kvm_gfn_range *range) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + + args->unreliable = true; +} +static inline unsigned long kvm_young_bitmap_offset (struct kvm_gfn_range *range, +gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +return hva_to_gfn_memslot (args->end -1 , range->slot) - gfn; +} +static inline void kvm_gfn_record_young (struct kvm_gfn_range *range,gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +WARN_ON_ONCE (gfn < range->start || gfn >= range->end); +if (args->bitmap) +__set_bit (kvm_young_bitmap_offset (range, gfn), args->bitmap); +} +static inline bool kvm_gfn_should_age (struct kvm_gfn_range *range,gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +WARN_ON_ONCE (gfn < range->start || gfn >= range->end); +if (args->bitmap) +return test_bit (kvm_young_bitmap_offset (range, gfn), + args->bitmap); +return true; +}#endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING virt/kvm/kvm_main.c +22
-12@@ -550,6 +550,7 @@ struct kvm_mmu_notifier_range {on_lock_fn_t on_lock;bool flush_on_ret;bool may_block; +bool lockless; };/* @@ -598,6 +599,8 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,struct kvm_memslots *slots;int i, idx; +BUILD_BUG_ON (sizeof (gfn_range.arg) !=sizeof (gfn_range.arg.pte)); +if (WARN_ON_ONCE (range->end <= range->start))return r;@@ -637,15 +640,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless;if (!r.found_memslot) { r.found_memslot = true; -KVM_MMU_LOCK (kvm); -if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm); - -if (IS_KVM_NULL_FN (range->handler)) -break ; +if (!range->lockless) { +KVM_MMU_LOCK (kvm); +if (!IS_KVM_NULL_FN (range->on_lock)) + range->on_lock (kvm); + +if (IS_KVM_NULL_FN (range->handler)) +break ; + } } r.ret |= range->handler (kvm, &gfn_range); }@@ -654,7 +660,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,if (range->flush_on_ret && r.ret)kvm_flush_remote_tlbs (kvm); -if (r.found_memslot) +if (r.found_memslot && !range->lockless)KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);@@ -682,19 +688,24 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,return __kvm_handle_hva_range (kvm, &range).ret; } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler) +static __always_inline int kvm_handle_hva_range_no_flush( +struct mmu_notifier *mn, +unsigned long start, +unsigned long end, +gfn_handler_t handler, +union kvm_mmu_notifier_arg arg, +bool lockless) {struct kvm *kvm =mmu_notifier_to_kvm (mn);const struct kvm_mmu_notifier_range range = { .start = start, .end = end, .handler = handler, + .arg = arg, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = lockless, };return __kvm_handle_hva_range (kvm, &range).ret;@@ -909,15 +920,36 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, kvm_age_gfn); } -static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, · line 928
Quoted 10 lines
virt/kvm/kvm_main.c +6
-4-struct mm_struct *mm, - unsigned long start, - unsigned long end, - unsigned long *bitmap) +static int kvm_mmu_notifier_test_clear_young (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start, +unsigned long end, +unsigned long *bitmap, +bool clear)
Perhaps pass in the callback (kvm_test_age_gfn/kvm_age_gfn) instead of true/false to avoid the naked booleans at the callsites?
Quoted 7 lines
virt/kvm/kvm_main.c +5
-1{ -trace_kvm_age_hva (start, end); +if (kvm_arch_prepare_bitmap_age (mn)) { +struct test_clear_young_metadata args = { + .bitmap = bitmap, + .end = end, + .unreliable = false,
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, · line 944
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ }; +union kvm_mmu_notifier_arg arg = { + .metadata = &args + }; +bool young; + + young =kvm_handle_hva_range_no_flush ( + mn, start, end, + clear ? kvm_age_gfn : kvm_test_age_gfn, + arg, true);
I suspect the end result will be cleaner we make all architectures lockless. i.e. Move the mmu_lock acquire/release into the architecture-specific code.
This could result in more acquire/release calls (one per memslot that overlaps the provided range) but that should be a single memslot in the majority of cases I think?
Then unconditionally pass in the metadata structure.
Then you don't need any special casing for the fast path / bitmap path. The only thing needed is to figure out whether to return MMU_NOTIFIER_YOUNG vs MMU_NOTIFIER_YOUNG_LOOK_AROUND and that can be plumbed via test_clear_young_metadata or by changing gfn_handler_t to return an int instead of a bool.
Quoted 13 lines
virt/kvm/kvm_main.c +7
-1+ +kvm_arch_finish_bitmap_age (mn); -/* We don't support bitmaps. Don't test or clear anything. */ +if (!args.unreliable) +return young ?MMU_NOTIFIER_YOUNG_FAST :0 ; + } + +/* A bitmap was passed but the architecture doesn't support bitmaps */ if (bitmap)return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ;
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, · line 971
Quoted 7 lines
virt/kvm/kvm_main.c +3
-1@@ -934,7 +966,21 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,* cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ -return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn); +return kvm_handle_hva_range_no_flush ( + mn, start, end, clear ? kvm_age_gfn : kvm_test_age_gfn, +KVM_MMU_NOTIFIER_NO_ARG , false);
Should this return MMU_NOTIFIER_YOUNG explicitly? This code is assuming MMU_NOTIFIER_YOUNG == (int)true.
Quoted 33 lines
virt/kvm/kvm_main.c +14
-6+} + +static int kvm_mmu_notifier_clear_young (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start, +unsigned long end, +unsigned long *bitmap) +{ +trace_kvm_age_hva (start, end); + +return kvm_mmu_notifier_test_clear_young (mn, mm, start, end, bitmap, + true); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,@@ -945,12 +991,8 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, {trace_kvm_test_age_hva (start, end); -/* We don't support bitmaps. Don't test or clear anything. */ -if (bitmap) -return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ; - -return kvm_handle_hva_range_no_flush (mn, start, end, - kvm_test_age_gfn); +return kvm_mmu_notifier_test_clear_young (mn, mm, start, end, bitmap, + false); } static void kvm_mmu_notifier_release(struct mmu_notifier *mn,Signature · 1 line
2.44.0.478.gd926399ef9-goog
Re: [PATCH v3 3/7] KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young
Quoted 8 lines · David Matlack
Quoted 5 lines · James Houghton
Add
kvm_arch_prepare_bitmap_age()for architectures to indiciate that they support bitmap-based aging inkvm_mmu_notifier_test_clear_young()and that they do not need KVM to grab the MMU lock for writing. This function allows architectures to do other locking or other preparatory work that it needs.
Quoted 4 lines
There's a lot going on here. I know it's extra work but I think the series would be easier to understand and simplify if you introduced the KVM support for lockless test/clear_young() first, and then introduce support for the bitmap-based look-around.
Yeah I think this is the right thing to do. Thanks.
Quoted 7 lines
Specifically:
1. Make all test/clear_young() notifiers lockless. i.e. Move the
mmu_lockinto the architecture-specific code (kvm_age_gfn()andkvm_test_age_gfn()).
Quoted 8 lines
2. Convert KVM/x86's kvm_{test,}_age_gfn() to be lockless for the TDP MMU.
4. Convert KVM/arm64's kvm_{test,}_age_gfn() to hold the
mmu_lockin read-mode.5. Add bitmap-based look-around support to KVM/x86 and KVM/arm64
(probably 2-3 patches).
This all sounds good to me. Thanks for laying it out for me -- this should be a lot simpler.
Quoted 29 lines
Quoted 29 lines
If an architecture does not implement
kvm_arch_prepare_bitmap_age()or is unable to do bitmap-based aging at runtime (and marks the bitmap as unreliable):
1. If a bitmap was provided, we inform the caller that the bitmap is unreliable (MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE).
2. If a bitmap was not provided, fall back to the old logic.Also add logic for architectures to easily use the provided bitmap if they are able. The expectation is that the architecture's implementation of
kvm_gfn_test_age()will usekvm_gfn_record_young(), andkvm_gfn_age()will usekvm_gfn_should_age().Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- include/linux/kvm_host.h | 60 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 92 +++++++++++++++++++++++++++++----------- 2 files changed, 127 insertions(+), 25 deletions(-)include/linux/kvm_host.h +0
-0@@ -1992,6 +1992,26 @@ extern const struct _kvm_stats_desc kvm_vm_stats_desc[];extern const struct kvm_stats_header kvm_vcpu_stats_header;extern const struct _kvm_stats_desc kvm_vcpu_stats_desc[];
Quoted 16 lines
include/linux/kvm_host.h extern const struct _kvm_stats_desc kvm_vm_stats_desc[]; · line 2006
Quoted 12 lines
include/linux/kvm_host.h +12
-0+/* +* Architectures that support using bitmaps for kvm_age_gfn() and +* kvm_test_age_gfn should return true for kvm_arch_prepare_bitmap_age() +* and do any work they need to prepare. The subsequent walk will not +* automatically grab the KVM MMU lock, so some architectures may opt +* to grab it. +* +* If true is returned, a subsequent call to kvm_arch_finish_bitmap_age() is +* guaranteed. +*/ +#ifndef kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn)I find the name of these architecture callbacks misleading/confusing. The lockless path is used even when a bitmap is not provided. i.e. bitmap can be NULL in between kvm_arch_prepare/finish_bitmap_age().
Yes. I am really terrible at picking names.... I'm happy to just nix this, following your other suggestions.
Quoted 12 lines
include/linux/kvm_host.h extern const struct _kvm_stats_desc kvm_vm_stats_desc[]; · line 2013
Quoted 8 lines
include/linux/kvm_host.h +7
-0+{ +return false; +} +#endif +#ifndef kvm_arch_finish_bitmap_age +static inline void kvm_arch_finish_bitmap_age (struct mmu_notifier *mn) {} +#endif
kvm_arch_finish_bitmap_age()seems unnecessary. I think the KVM/arm64 code could acquire/release themmu_lockin read-mode inkvm_test_age_gfn()andkvm_age_gfn()right?
Yes you're right, except that the way it is now, we only lock/unlock once for the notifier instead of once for each overlapping memslot, but that's not an issue, as you mention below.
Quoted 9 lines
Quoted 9 lines
include/linux/kvm_host.h +1
-0+#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -2076,9 +2096,16 @@ static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm,return READ_ONCE (kvm->mmu_invalidate_seq) != mmu_seq; }
Quoted 5 lines
include/linux/kvm_host.h static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, · line 2101
Quoted 3 lines
include/linux/kvm_host.h +3
-0+struct test_clear_young_metadata { +unsigned long *bitmap; +unsigned long bitmap_offset_end;
bitmap_offset_endis unused.
Indeed, sorry about that.
Quoted 10 lines
include/linux/kvm_host.h static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, · line 2108
Quoted 8 lines
include/linux/kvm_host.h +4
-0+unsigned long end; +bool unreliable; +};union kvm_mmu_notifier_arg {pte_t pte;unsigned long attributes; +struct test_clear_young_metadata *metadata;nit: Maybe s/metadata/test_clear_young/ ?
Yes, that's better.
Quoted 12 lines
include/linux/kvm_host.h struct kvm_gfn_range { · line 2117
Quoted 9 lines
include/linux/kvm_host.h +1
-0};struct kvm_gfn_range {@@ -2087,11 +2114,44 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless;Please document this as it's somewhat subtle. A reader might think this implies the entire operation runs without taking the
mmu_lock.
Will do, and I'll improve the comments for the other "lockless" variables. (In fact, it might be better to rename/adjust this one to "mmu_lock_taken" instead -- it's a little more obvious what that means.)
Quoted 129 lines
Quoted 129 lines
include/linux/kvm_host.h +32
-0};bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range); + +static inline void kvm_age_set_unreliable (struct kvm_gfn_range *range) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + + args->unreliable = true; +} +static inline unsigned long kvm_young_bitmap_offset (struct kvm_gfn_range *range, +gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +return hva_to_gfn_memslot (args->end -1 , range->slot) - gfn; +} +static inline void kvm_gfn_record_young (struct kvm_gfn_range *range,gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +WARN_ON_ONCE (gfn < range->start || gfn >= range->end); +if (args->bitmap) +__set_bit (kvm_young_bitmap_offset (range, gfn), args->bitmap); +} +static inline bool kvm_gfn_should_age (struct kvm_gfn_range *range,gfn_t gfn) +{ +struct test_clear_young_metadata *args = range->arg.metadata; + +WARN_ON_ONCE (gfn < range->start || gfn >= range->end); +if (args->bitmap) +return test_bit (kvm_young_bitmap_offset (range, gfn), + args->bitmap); +return true; +}#endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING virt/kvm/kvm_main.c +22
-11@@ -550,6 +550,7 @@ struct kvm_mmu_notifier_range {on_lock_fn_t on_lock;bool flush_on_ret;bool may_block; +bool lockless; };/* @@ -598,6 +599,8 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,struct kvm_memslots *slots;int i, idx; +BUILD_BUG_ON (sizeof (gfn_range.arg) !=sizeof (gfn_range.arg.pte)); +if (WARN_ON_ONCE (range->end <= range->start))return r;@@ -637,15 +640,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless;if (!r.found_memslot) { r.found_memslot = true; -KVM_MMU_LOCK (kvm); -if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm); - -if (IS_KVM_NULL_FN (range->handler)) -break ; +if (!range->lockless) { +KVM_MMU_LOCK (kvm); +if (!IS_KVM_NULL_FN (range->on_lock)) + range->on_lock (kvm); + +if (IS_KVM_NULL_FN (range->handler)) +break ; + } } r.ret |= range->handler (kvm, &gfn_range); }@@ -654,7 +660,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,if (range->flush_on_ret && r.ret)kvm_flush_remote_tlbs (kvm); -if (r.found_memslot) +if (r.found_memslot && !range->lockless)KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);@@ -682,19 +688,24 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,return __kvm_handle_hva_range (kvm, &range).ret; } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler) +static __always_inline int kvm_handle_hva_range_no_flush( +struct mmu_notifier *mn, +unsigned long start, +unsigned long end, +gfn_handler_t handler, +union kvm_mmu_notifier_arg arg, +bool lockless) {struct kvm *kvm =mmu_notifier_to_kvm (mn);const struct kvm_mmu_notifier_range range = { .start = start, .end = end, .handler = handler, + .arg = arg, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = lockless, };return __kvm_handle_hva_range (kvm, &range).ret;@@ -909,15 +920,36 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, kvm_age_gfn); }
Quoted 14 lines
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, · line 928
Quoted 11 lines
virt/kvm/kvm_main.c +6
-5-static int kvm_mmu_notifier_clear_young (struct mmu_notifier *mn, -struct mm_struct *mm, -unsigned long start, -unsigned long end, -unsigned long *bitmap) +static int kvm_mmu_notifier_test_clear_young (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start, +unsigned long end, +unsigned long *bitmap, +bool clear)Perhaps pass in the callback (kvm_test_age_gfn/kvm_age_gfn) instead of true/false to avoid the naked booleans at the callsites?
Will do. Thank you.
Quoted 27 lines
Quoted 8 lines
virt/kvm/kvm_main.c +5
-1{ -trace_kvm_age_hva (start, end); +if (kvm_arch_prepare_bitmap_age (mn)) { +struct test_clear_young_metadata args = { + .bitmap = bitmap, + .end = end, + .unreliable = false,virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, · line 944
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ }; +union kvm_mmu_notifier_arg arg = { + .metadata = &args + }; +bool young; + + young =kvm_handle_hva_range_no_flush ( + mn, start, end, + clear ? kvm_age_gfn : kvm_test_age_gfn, + arg, true);I suspect the end result will be cleaner we make all architectures lockless. i.e. Move the
mmu_lockacquire/release into the architecture-specific code.This could result in more acquire/release calls (one per memslot that overlaps the provided range) but that should be a single memslot in the majority of cases I think?
Quoted 7 lines
Then unconditionally pass in the metadata structure.
Then you don't need any special casing for the fast path / bitmap path. The only thing needed is to figure out whether to return
MMU_NOTIFIER_YOUNGvsMMU_NOTIFIER_YOUNG_LOOK_AROUNDand that can be plumbed viatest_clear_young_metadataor by changinggfn_handler_tto return an int instead of a bool.
Yes I think this simplification is a great idea. I agree that usually there will only be one memslot that overlaps a virtual address range in practice (MIN_LRU_BATCH is BITS_PER_LONG), so the theoretical additional locking/unlocking shouldn't be an issue.
Quoted 13 lines
Quoted 13 lines
virt/kvm/kvm_main.c +7
-1+ +kvm_arch_finish_bitmap_age (mn); -/* We don't support bitmaps. Don't test or clear anything. */ +if (!args.unreliable) +return young ?MMU_NOTIFIER_YOUNG_FAST :0 ; + } + +/* A bitmap was passed but the architecture doesn't support bitmaps */ if (bitmap)return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE ;
Quoted 11 lines
virt/kvm/kvm_main.c static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, · line 971
Quoted 8 lines
virt/kvm/kvm_main.c +3
-1@@ -934,7 +966,21 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,* cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ -return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn); +return kvm_handle_hva_range_no_flush ( + mn, start, end, clear ? kvm_age_gfn : kvm_test_age_gfn, +KVM_MMU_NOTIFIER_NO_ARG , false);Should this return
MMU_NOTIFIER_YOUNGexplicitly? This code is assumingMMU_NOTIFIER_YOUNG== (int)true.
Yes.
Thank you for all the feedback!
PATCH v3 4/7 KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask
In reply to James HoughtonFrom: Yu Zhao <[email protected]>
tdp_mmu_enabled and shadow_accessed_mask are needed to implement kvm_arch_prepare_bitmap_age().
--- arch/x86/include/asm/kvm_host.h | 6 ++++++ arch/x86/kvm/mmu.h | 6 ------ arch/x86/kvm/mmu/spte.h | 1 - 3 files changed, 6 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +6 -0
@@ -1847,6 +1847,7 @@ struct kvm_arch_async_pf {extern u32 __read_mostly kvm_nr_uret_msrs;extern u64 __read_mostly host_efer; +extern u64 __read_mostly shadow_accessed_mask;extern bool __read_mostly allow_smaller_maxphyaddr;extern bool __read_mostly enable_apicv;extern struct kvm_x86_ops kvm_x86_ops;
@@ -1952,6 +1953,11 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin, bool mask);extern bool tdp_enabled; +#ifdef CONFIG_X86_64 +extern bool tdp_mmu_enabled; +#else +#define tdp_mmu_enabled false +#endif u64 vcpu_tsc_khz (struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu.h +0 -6
@@ -270,12 +270,6 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm)return smp_load_acquire (&kvm->arch.shadow_root_allocated); } -#ifdef CONFIG_X86_64 -extern bool tdp_mmu_enabled; -#else -#define tdp_mmu_enabled false -#endif -static inline bool kvm_memslots_have_rmaps (struct kvm *kvm) {return !tdp_mmu_enabled ||kvm_shadow_root_allocated (kvm);
arch/x86/kvm/mmu/spte.h +0 -1
@@ -154,7 +154,6 @@ extern u64 __read_mostly shadow_mmu_writable_mask;extern u64 __read_mostly shadow_nx_mask;extern u64 __read_mostly shadow_x_mask;/* mutual exclusive with nx_mask */ extern u64 __read_mostly shadow_user_mask; -extern u64 __read_mostly shadow_accessed_mask;extern u64 __read_mostly shadow_dirty_mask;extern u64 __read_mostly shadow_mmio_value;extern u64 __read_mostly shadow_mmio_mask;
Signature · 1 line
2.44.0.478.gd926399ef9-goog
PATCH v3 5/7 KVM: x86: Participate in bitmap-based PTE aging
In reply to James HoughtonOnly handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.
--- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +14 -0
@@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);*/ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +/* +* Indicate that we support bitmap-based aging when using the TDP MMU +* and the accessed bit is available in the TDP page tables. +* +* We have no other preparatory work to do here, so we do not need to +* redefine kvm_arch_finish_bitmap_age(). +*/ +return IS_ENABLED (CONFIG_X86_64 ) && tdp_mmu_enabled + && shadow_accessed_mask; +} +#endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/mmu/mmu.c +14 -2
@@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +if (range->lockless) { +kvm_age_set_unreliable (range); +return false; + } + young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range);
@@ -1687,8 +1693,14 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +if (range->lockless) { +kvm_age_set_unreliable (range); +return false; + } + young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range);
arch/x86/kvm/mmu/tdp_mmu.c +9 -1
@@ -1217,6 +1217,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter,if (!is_accessed_spte (iter->old_spte))return false; +if (!kvm_gfn_should_age (range, iter->gfn)) +return false; +if (spte_ad_enabled (iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte,
@@ -1250,7 +1253,12 @@ bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)static bool test_age_gfn (struct kvm *kvm,struct tdp_iter *iter,struct kvm_gfn_range *range) { -return is_accessed_spte (iter->old_spte); +bool young =is_accessed_spte (iter->old_spte); + +if (young) +kvm_gfn_record_young (range, iter->gfn); + +return young; }bool kvm_tdp_mmu_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range)
Signature · 1 line
2.44.0.478.gd926399ef9-goog
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 40 lines · James Houghton
Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes
mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +14
-0@@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);*/ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +/* +* Indicate that we support bitmap-based aging when using the TDP MMU +* and the accessed bit is available in the TDP page tables. +* +* We have no other preparatory work to do here, so we do not need to +* redefine kvm_arch_finish_bitmap_age(). +*/ +return IS_ENABLED (CONFIG_X86_64 ) && tdp_mmu_enabled + && shadow_accessed_mask; +} +#endif /* _ASM_X86_KVM_HOST_H */ arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · line 1681
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +5
-1@@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +if (range->lockless) { +kvm_age_set_unreliable (range); +return false; + }
If a VM has TDP MMU enabled, supports A/D bits, and is using nested virtualization, MGLRU will effectively be blind to all accesses made by the VM.
kvm_arch_prepare_bitmap_age() will return true indicating that the bitmap is supported. But then kvm_age_gfn() and kvm_test_age_gfn() will return false immediately and indicate the bitmap is unreliable because a shadow root is allocate. The notfier will then return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.
Looking at the callers, MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE is never consumed or used. So I think MGLRU will assume all memory is unaccessed?
One way to improve the situation would be to re-order the TDP MMU function first and return young instead of false, so that way MGLRU at least has visibility into accesses made by L1 (and L2 if EPT is disable in L2). But that still means MGLRU is blind to accesses made by L2.
What about grabbing the mmu_lock if there's a shadow root allocated and get rid of MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE altogether?
if (kvm_memslots_have_rmaps(kvm)) {
write_lock(&kvm->mmu_lock);
young |= kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
write_unlock(&kvm->mmu_lock);
}The TDP MMU walk would still be lockless. KVM only has to take the mmu_lock to collect accesses made by L2.
kvm_age_rmap() and kvm_test_age_rmap() will need to become bitmap-aware as well, but that seems relatively simple with the helper functions.
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 79 lines · David Matlack
Quoted 40 lines · James Houghton
Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes
mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +14
-0@@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);*/ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +/* +* Indicate that we support bitmap-based aging when using the TDP MMU +* and the accessed bit is available in the TDP page tables. +* +* We have no other preparatory work to do here, so we do not need to +* redefine kvm_arch_finish_bitmap_age(). +*/ +return IS_ENABLED (CONFIG_X86_64 ) && tdp_mmu_enabled + && shadow_accessed_mask; +} +#endif /* _ASM_X86_KVM_HOST_H */ arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · line 1681
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +5
-1@@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +if (range->lockless) { +kvm_age_set_unreliable (range); +return false; + }If a VM has TDP MMU enabled, supports A/D bits, and is using nested virtualization, MGLRU will effectively be blind to all accesses made by the VM.
kvm_arch_prepare_bitmap_age()will return true indicating that the bitmap is supported. But thenkvm_age_gfn()andkvm_test_age_gfn()will return false immediately and indicate the bitmap is unreliable because a shadow root is allocate. The notfier will then returnMMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.Looking at the callers,
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEis never consumed or used. So I think MGLRU will assume all memory is unaccessed?One way to improve the situation would be to re-order the TDP MMU function first and return young instead of false, so that way MGLRU at least has visibility into accesses made by L1 (and L2 if EPT is disable in L2). But that still means MGLRU is blind to accesses made by L2.
What about grabbing the
mmu_lockif there's a shadow root allocated and get rid ofMMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEaltogether?if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); young |= kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); }
Quoted 5 lines
The TDP MMU walk would still be lockless. KVM only has to take the
mmu_lockto collect accesses made by L2.
kvm_age_rmap()andkvm_test_age_rmap()will need to become bitmap-aware as well, but that seems relatively simple with the helper functions.
Wait, even simpler, just check kvm_memslots_have_rmaps() in kvm_arch_prepare_bitmap_age() and skip the shadow MMU when processing a bitmap request.
i.e.
static inline bool kvm_arch_prepare_bitmap_age(struct kvm *kvm, struct mmu_notifier *mn)
{
/*
* Indicate that we support bitmap-based aging when using the TDP MMU
* and the accessed bit is available in the TDP page tables.
*
* We have no other preparatory work to do here, so we do not need to
* redefine kvm_arch_finish_bitmap_age().
*/
return IS_ENABLED(CONFIG_X86_64)
&& tdp_mmu_enabled
&& shadow_accessed_mask
&& !kvm_memslots_have_rmaps(kvm);
}bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
bool young = false;
if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm))
young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
if (tdp_mmu_enabled)
young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
return young;
}bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
bool young = false;
if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm))
young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
if (tdp_mmu_enabled)
young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
return young;
}Sure this could race with the creation of a shadow root but so can the non-bitmap code.
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 57 lines · David Matlack
Quoted 55 lines · David Matlack
Quoted 40 lines · James Houghton
Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes
mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +14
-0@@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);*/ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +/* +* Indicate that we support bitmap-based aging when using the TDP MMU +* and the accessed bit is available in the TDP page tables. +* +* We have no other preparatory work to do here, so we do not need to +* redefine kvm_arch_finish_bitmap_age(). +*/ +return IS_ENABLED (CONFIG_X86_64 ) && tdp_mmu_enabled + && shadow_accessed_mask; +} +#endif /* _ASM_X86_KVM_HOST_H */ arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · line 1681
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +5
-1@@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +if (range->lockless) { +kvm_age_set_unreliable (range); +return false; + }If a VM has TDP MMU enabled, supports A/D bits, and is using nested virtualization, MGLRU will effectively be blind to all accesses made by the VM.
Quoted 5 lines
Quoted 5 lines
kvm_arch_prepare_bitmap_age()will return true indicating that the bitmap is supported. But thenkvm_age_gfn()andkvm_test_age_gfn()will return false immediately and indicate the bitmap is unreliable because a shadow root is allocate. The notfier will then returnMMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.
Ah no, I'm wrong here. Setting args.unreliable causes the notifier to return 0 instead of MMU_NOTIFIER_YOUNG_FAST. MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE is used for something else.
The control flow of all this and naming of functions and macros is overall confusing. args.unreliable and MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE for one. Also I now realize kvm_arch_prepare/finish_bitmap_age() are used even when the bitmap is _not_ provided, so those names are also misleading.
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 65 lines · David Matlack
Quoted 57 lines · David Matlack
Quoted 55 lines · David Matlack
Quoted 40 lines · James Houghton
Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes
mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +14
-0@@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);*/ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +/* +* Indicate that we support bitmap-based aging when using the TDP MMU +* and the accessed bit is available in the TDP page tables. +* +* We have no other preparatory work to do here, so we do not need to +* redefine kvm_arch_finish_bitmap_age(). +*/ +return IS_ENABLED (CONFIG_X86_64 ) && tdp_mmu_enabled + && shadow_accessed_mask; +} +#endif /* _ASM_X86_KVM_HOST_H */ arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · line 1681
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +5
-1@@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +if (range->lockless) { +kvm_age_set_unreliable (range); +return false; + }If a VM has TDP MMU enabled, supports A/D bits, and is using nested virtualization, MGLRU will effectively be blind to all accesses made by the VM.
Quoted 5 lines
Quoted 5 lines
kvm_arch_prepare_bitmap_age()will return true indicating that the bitmap is supported. But thenkvm_age_gfn()andkvm_test_age_gfn()will return false immediately and indicate the bitmap is unreliable because a shadow root is allocate. The notfier will then returnMMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.
Quoted 3 lines
Ah no, I'm wrong here. Setting args.unreliable causes the notifier to return 0 instead of
MMU_NOTIFIER_YOUNG_FAST.MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEis used for something else.
Nope, wrong again. Just ignore me while I try to figure out how this actually works :)
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
In reply to David MatlackQuoted 122 lines · David Matlack
Quoted 79 lines · David Matlack
Quoted 40 lines · James Houghton
Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes
mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +14
-0@@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);*/ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +/* +* Indicate that we support bitmap-based aging when using the TDP MMU +* and the accessed bit is available in the TDP page tables. +* +* We have no other preparatory work to do here, so we do not need to +* redefine kvm_arch_finish_bitmap_age(). +*/ +return IS_ENABLED (CONFIG_X86_64 ) && tdp_mmu_enabled + && shadow_accessed_mask; +} +#endif /* _ASM_X86_KVM_HOST_H */ arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · line 1681
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +5
-1@@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +if (range->lockless) { +kvm_age_set_unreliable (range); +return false; + }If a VM has TDP MMU enabled, supports A/D bits, and is using nested virtualization, MGLRU will effectively be blind to all accesses made by the VM.
kvm_arch_prepare_bitmap_age()will return true indicating that the bitmap is supported. But thenkvm_age_gfn()andkvm_test_age_gfn()will return false immediately and indicate the bitmap is unreliable because a shadow root is allocate. The notfier will then returnMMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE.Looking at the callers,
MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEis never consumed or used. So I think MGLRU will assume all memory is unaccessed?One way to improve the situation would be to re-order the TDP MMU function first and return young instead of false, so that way MGLRU at least has visibility into accesses made by L1 (and L2 if EPT is disable in L2). But that still means MGLRU is blind to accesses made by L2.
What about grabbing the
mmu_lockif there's a shadow root allocated and get rid ofMMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLEaltogether?if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); young |= kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); }Quoted 5 lines
The TDP MMU walk would still be lockless. KVM only has to take the
mmu_lockto collect accesses made by L2.
kvm_age_rmap()andkvm_test_age_rmap()will need to become bitmap-aware as well, but that seems relatively simple with the helper functions.Wait, even simpler, just check
kvm_memslots_have_rmaps()inkvm_arch_prepare_bitmap_age()and skip the shadow MMU when processing a bitmap request.i.e.
static inline bool
kvm_arch_prepare_bitmap_age(struct kvm *kvm, struct mmu_notifier *mn){ /* * Indicate that we support bitmap-based aging when using the TDP MMU * and the accessed bit is available in the TDP page tables. * * We have no other preparatory work to do here, so we do not need to * redefine kvm_arch_finish_bitmap_age(). */ return IS_ENABLED(CONFIG_X86_64) && tdp_mmu_enabled && shadow_accessed_mask && !kvm_memslots_have_rmaps(kvm); }bool
kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ bool young = false; if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }bool
kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
Quoted 10 lines
{
bool young = false;
if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm))
young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
if (tdp_mmu_enabled)
young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
return young;Yeah I think this is the right thing to do. Given your other suggestions (on patch 3), I think this will look something like this -- let me know if I've misunderstood something:
bool check_rmap = !bitmap && kvm_memslot_have_rmaps(kvm);
if (check_rmap)
KVM_MMU_LOCK(kvm);
rcu_read_lock(); // perhaps only do this when we don't take the MMU lock?
if (check_rmap)
kvm_handle_gfn_range(/* ... */ kvm_test_age_rmap)
if (tdp_mmu_enabled)
kvm_tdp_mmu_test_age_gfn() // modified to be RCU-safe
rcu_read_unlock();
if (check_rmap)
KVM_MMU_UNLOCK(kvm);
Quoted 4 lines
}
Sure this could race with the creation of a shadow root but so can the non-bitmap code.
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 42 lines · James Houghton
Quoted 15 lines · David Matlack
On 2024-04-11 10:08 AM, David Matlack wrote:
boolkvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ bool young = false; if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }bool
kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)Quoted 10 lines
{ bool young = false; if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); return young;Yeah I think this is the right thing to do. Given your other suggestions (on patch 3), I think this will look something like this -- let me know if I've misunderstood something:
bool
check_rmap= !bitmap &&kvm_memslot_have_rmaps(kvm);if (
check_rmap)
KVM_MMU_LOCK(kvm);
rcu_read_lock(); // perhaps only do this when we don't take the MMU lock?if (
check_rmap)
kvm_handle_gfn_range(/* ... */kvm_test_age_rmap)
Quoted 6 lines
if (
tdp_mmu_enabled)
kvm_tdp_mmu_test_age_gfn()// modified to be RCU-safe
rcu_read_unlock();
if (check_rmap)
KVM_MMU_UNLOCK(kvm);
I was thinking a little different. If you follow my suggestion to first make the TDP MMU aging lockless, you'll end up with something like this prior to adding bitmap support (note: the comments are just for demonstrative purposes):
bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
bool young = false;
/* Shadow MMU aging holds write-lock. */
if (kvm_memslots_have_rmaps(kvm)) {
write_lock(&kvm->mmu_lock);
young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
write_unlock(&kvm->mmu_lock);
}
/* TDM MMU aging is lockless. */
if (tdp_mmu_enabled)
young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
return young;
}Then when you add bitmap support it would look something like this:
bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
unsigned long *bitmap = range->arg.metadata->bitmap;
bool young = false;
/* SHadow MMU aging holds write-lock and does not support bitmap. */
if (kvm_memslots_have_rmaps(kvm) && !bitmap) {
write_lock(&kvm->mmu_lock);
young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
write_unlock(&kvm->mmu_lock);
}
/* TDM MMU aging is lockless and supports bitmap. */
if (tdp_mmu_enabled)
young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
return young;
}rcu_read_lock/unlock() would be called in kvm_tdp_mmu_age_gfn_range().
That brings up a question I've been wondering about. If KVM only advertises support for the bitmap lookaround when shadow roots are not allocated, does that mean MGLRU will be blind to accesses made by L2 when nested virtualization is enabled? And does that mean the Linux MM will think all L2 memory is cold (i.e. good candidate for swapping) because it isn't seeing accesses made by L2?
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 95 lines · David Matlack
Quoted 42 lines · James Houghton
Quoted 15 lines · David Matlack
On 2024-04-11 10:08 AM, David Matlack wrote:
boolkvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ bool young = false; if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }bool
kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)Quoted 10 lines
{ bool young = false; if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); return young;Yeah I think this is the right thing to do. Given your other suggestions (on patch 3), I think this will look something like this -- let me know if I've misunderstood something:
bool
check_rmap= !bitmap &&kvm_memslot_have_rmaps(kvm);if (
check_rmap)
KVM_MMU_LOCK(kvm);
rcu_read_lock(); // perhaps only do this when we don't take the MMU lock?if (
check_rmap)
kvm_handle_gfn_range(/* ... */kvm_test_age_rmap)Quoted 6 lines
if (
tdp_mmu_enabled)
kvm_tdp_mmu_test_age_gfn()// modified to be RCU-safe
rcu_read_unlock();
if (check_rmap)
KVM_MMU_UNLOCK(kvm);I was thinking a little different. If you follow my suggestion to first make the TDP MMU aging lockless, you'll end up with something like this prior to adding bitmap support (note: the comments are just for demonstrative purposes):
bool
kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ bool young = false; /* Shadow MMU aging holds write-lock. */ if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); } /* TDM MMU aging is lockless. */ if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }Then when you add bitmap support it would look something like this:
bool
kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ unsigned long *bitmap = range->arg.metadata->bitmap; bool young = false; /* SHadow MMU aging holds write-lock and does not support bitmap. */ if (kvm_memslots_have_rmaps(kvm) && !bitmap) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); } /* TDM MMU aging is lockless and supports bitmap. */ if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }
Quoted 1 line
rcu_read_lock/unlock() would be called in
kvm_tdp_mmu_age_gfn_range().
Oh yes this is a lot better. I hope I would have seen this when it came time to actually update this patch. Thanks.
Quoted 7 lines
That brings up a question I've been wondering about. If KVM only advertises support for the bitmap lookaround when shadow roots are not allocated, does that mean MGLRU will be blind to accesses made by L2 when nested virtualization is enabled? And does that mean the Linux MM will think all L2 memory is cold (i.e. good candidate for swapping) because it isn't seeing accesses made by L2?
Yes, I think so (for both questions). That's better than KVM not participating in MGLRU aging at all, which is the case today (IIUC -- also ignoring the case where KVM accesses guest memory directly). We could have MGLRU always invoke the mmu notifiers, but frequently taking the MMU lock for writing might be worse than evicting when we shouldn't. Maybe Yu tried this at some point, but I can't find any results for this.
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 110 lines · James Houghton
Quoted 95 lines · David Matlack
Quoted 42 lines · James Houghton
Quoted 15 lines · David Matlack
On 2024-04-11 10:08 AM, David Matlack wrote:
boolkvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ bool young = false; if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }bool
kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)Quoted 10 lines
{ bool young = false; if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); return young;Yeah I think this is the right thing to do. Given your other suggestions (on patch 3), I think this will look something like this -- let me know if I've misunderstood something:
bool
check_rmap= !bitmap &&kvm_memslot_have_rmaps(kvm);if (
check_rmap)
KVM_MMU_LOCK(kvm);
rcu_read_lock(); // perhaps only do this when we don't take the MMU lock?if (
check_rmap)
kvm_handle_gfn_range(/* ... */kvm_test_age_rmap)Quoted 6 lines
if (
tdp_mmu_enabled)
kvm_tdp_mmu_test_age_gfn()// modified to be RCU-safe
rcu_read_unlock();
if (check_rmap)
KVM_MMU_UNLOCK(kvm);I was thinking a little different. If you follow my suggestion to first make the TDP MMU aging lockless, you'll end up with something like this prior to adding bitmap support (note: the comments are just for demonstrative purposes):
bool
kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ bool young = false; /* Shadow MMU aging holds write-lock. */ if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); } /* TDM MMU aging is lockless. */ if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }Then when you add bitmap support it would look something like this:
bool
kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range){ unsigned long *bitmap = range->arg.metadata->bitmap; bool young = false; /* SHadow MMU aging holds write-lock and does not support bitmap. */ if (kvm_memslots_have_rmaps(kvm) && !bitmap) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); } /* TDM MMU aging is lockless and supports bitmap. */ if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; }Quoted 1 line
rcu_read_lock/unlock() would be called in
kvm_tdp_mmu_age_gfn_range().Oh yes this is a lot better. I hope I would have seen this when it came time to actually update this patch. Thanks.
Quoted 7 lines
That brings up a question I've been wondering about. If KVM only advertises support for the bitmap lookaround when shadow roots are not allocated, does that mean MGLRU will be blind to accesses made by L2 when nested virtualization is enabled? And does that mean the Linux MM will think all L2 memory is cold (i.e. good candidate for swapping) because it isn't seeing accesses made by L2?
Quoted 7 lines
Yes, I think so (for both questions). That's better than KVM not participating in MGLRU aging at all, which is the case today (IIUC -- also ignoring the case where KVM accesses guest memory directly). We could have MGLRU always invoke the mmu notifiers, but frequently taking the MMU lock for writing might be worse than evicting when we shouldn't. Maybe Yu tried this at some point, but I can't find any results for this.
No, in this case only the fast path (page table scanning) is disabled. MGLRU still sees the A-bit from L2 using the rmap, i.e., the slow path calling folio_check_references().
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
In reply to James HoughtonQuoted 3 lines · James Houghton
Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes
mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.
I think this patch will trigger a lockdep assert in
kvm_tdp_mmu_age_gfn_range
kvm_tdp_mmu_handle_gfn
for_each_tdp_mmu_root
__for_each_tdp_mmu_root
kvm_lockdep_assert_mmu_lock_held... because it walks tdp_mmu_roots without holding mmu_lock.
Yu's patch[1] added a lockless walk to the TDP MMU. We'd need something similar here and also update the comment above tdp_mmu_roots describing how tdp_mmu_roots can be read locklessly.
Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Quoted 8 lines · David Matlack
Quoted 3 lines · James Houghton
Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes
mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable.I think this patch will trigger a lockdep assert in
Quoted 7 lines
kvm_tdp_mmu_age_gfn_rangekvm_tdp_mmu_handle_gfn for_each_tdp_mmu_root __for_each_tdp_mmu_root kvm_lockdep_assert_mmu_lock_held... because it walks
tdp_mmu_rootswithout holdingmmu_lock.
Indeed, thanks. I'll make sure to build with CONFIG_LOCKDEP for the future versions and check for errors.
Quoted 4 lines
Yu's patch[1] added a lockless walk to the TDP MMU. We'd need something similar here and also update the comment above
tdp_mmu_rootsdescribing howtdp_mmu_rootscan be read locklessly.
I'll add the macro / function to do the lockless walk of tdp_mmu_roots and explain why it is safe. Thanks for pointing out this big mistake.
Quoted 1 line
PATCH v3 6/7 KVM: arm64: Participate in bitmap-based PTE aging
In reply to James HoughtonParticipate in bitmap-based aging while grabbing the KVM MMU lock for reading. Ideally we wouldn't need to grab this lock at all, but that would require a more intrustive and risky change. Also pass KVM_PGTABLE_WALK_SHARED, as this software walker is safe to run in parallel with other walkers.
It is safe only to grab the KVM MMU lock for reading as the kvm_pgtable is destroyed while holding the lock for writing, and freeing of the page table pages is either done while holding the MMU lock for writing or after an RCU grace period.
When mkold == false, record the young pages in the passed-in bitmap.
When mkold == true, only age the pages that need aging according to the passed-in bitmap.
--- arch/arm64/include/asm/kvm_host.h | 5 +++++ arch/arm64/include/asm/kvm_pgtable.h | 4 +++- arch/arm64/kvm/hyp/pgtable.c | 21 ++++++++++++++------- arch/arm64/kvm/mmu.c | 23 +++++++++++++++++++++-- 4 files changed, 43 insertions(+), 10 deletions(-)
arch/arm64/include/asm/kvm_host.h +5 -0
@@ -1331,4 +1331,9 @@ bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu); (get_idreg_field ((kvm), id, fld) >=expand_field_sign (id, fld, min) && \ get_idreg_field((kvm), id, fld) <= expand_field_sign(id, fld, max)) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn); +#define kvm_arch_finish_bitmap_age kvm_arch_finish_bitmap_age +void kvm_arch_finish_bitmap_age (struct mmu_notifier *mn); +#endif /* __ARM64_KVM_HOST_H__ */
arch/arm64/include/asm/kvm_pgtable.h +3 -1
@@ -644,6 +644,7 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr);* @addr: Intermediate physical address to identify the page-table entry. * @size: Size of the address range to visit. * @mkold: True if the access flag should be cleared. +* @range: The kvm_gfn_range that is being used for the memslot walker. * * The offset of @addr within a page is ignored. *
@@ -657,7 +658,8 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr);* Return: True if any of the visited PTEs had the access flag set. */ bool kvm_pgtable_stage2_test_clear_young (struct kvm_pgtable *pgt,u64 addr, -u64 size,bool mkold); +u64 size,bool mkold, +struct kvm_gfn_range *range);/** * kvm_pgtable_stage2_relax_perms() - Relax the permissions enforced by a
arch/arm64/kvm/hyp/pgtable.c +14 -7
@@ -1281,6 +1281,7 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr) }struct stage2_age_data { +struct kvm_gfn_range *range;bool mkold;bool young; };
@@ -1290,20 +1291,24 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, {kvm_pte_t new = ctx->old & ~KVM_PTE_LEAF_ATTR_LO_S2_AF ;struct stage2_age_data *data = ctx->arg; +gfn_t gfn = ctx->addr /PAGE_SIZE ;if (!kvm_pte_valid (ctx->old) || new == ctx->old)return 0 ; data->young = true; +/* -* stage2_age_walker() is always called while holding the MMU lock for -* write, so this will always succeed. Nonetheless, this deliberately -* follows the race detection pattern of the other stage-2 walkers in -* case the locking mechanics of the MMU notifiers is ever changed. +* stage2_age_walker() may not be holding the MMU lock for write, so +* follow the race detection pattern of the other stage-2 walkers. */ -if (data->mkold && !stage2_try_set_pte (ctx, new)) -return -EAGAIN ; +if (data->mkold) { +if (kvm_gfn_should_age (data->range, gfn) && + !stage2_try_set_pte (ctx, new)) +return -EAGAIN ; + }else +kvm_gfn_record_young (data->range, gfn);/* * "But where's the TLBI?!", you scream.
@@ -1315,10 +1320,12 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, }bool kvm_pgtable_stage2_test_clear_young (struct kvm_pgtable *pgt,u64 addr, -u64 size,bool mkold) +u64 size,bool mkold, +struct kvm_gfn_range *range) {struct stage2_age_data data = { .mkold = mkold, + .range = range, };struct kvm_pgtable_walker walker = { .cb = stage2_age_walker,
arch/arm64/kvm/mmu.c +21 -2
@@ -1802,6 +1802,25 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range)return false; } +bool kvm_arch_prepare_bitmap_age (struct mmu_notifier *mn) +{ +struct kvm *kvm =mmu_notifier_to_kvm (mn); + +/* +* We need to hold the MMU lock for reading to prevent page tables +* from being freed underneath us. +*/ +read_lock (&kvm->mmu_lock); +return true; +} + +void kvm_arch_finish_bitmap_age (struct mmu_notifier *mn) +{ +struct kvm *kvm =mmu_notifier_to_kvm (mn); + +read_unlock (&kvm->mmu_lock); +} +bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {u64 size = (range->end - range->start) <<PAGE_SHIFT ;
@@ -1811,7 +1830,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)return kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, range->start <<PAGE_SHIFT , - size, true); + size, true, range); }bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range)
@@ -1823,7 +1842,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)return kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, range->start <<PAGE_SHIFT , - size, false); + size, false, range); }phys_addr_t kvm_mmu_get_httbr (void )
Signature · 1 line
2.44.0.478.gd926399ef9-goog
Re: [PATCH v3 6/7] KVM: arm64: Participate in bitmap-based PTE aging
Quoted 4 lines · James Houghton
Participate in bitmap-based aging while grabbing the KVM MMU lock for reading. Ideally we wouldn't need to grab this lock at all, but that would require a more intrustive and risky change.
^^^^^^^^^^ intrusive
This sounds subjective -- I'd just present the challenges and let reviewers make their own judgements.
Quoted 9 lines
Also pass
KVM_PGTABLE_WALK_SHARED, as this software walker is safe to run in parallel with other walkers.It is safe only to grab the KVM MMU lock for reading as the
kvm_pgtableis destroyed while holding the lock for writing, and freeing of the page table pages is either done while holding the MMU lock for writing or after an RCU grace period.
Quoted 6 lines
When mkold == false, record the young pages in the passed-in bitmap.
When mkold == true, only age the pages that need aging according to the passed-in bitmap.
Suggested-by Yu Zhao <[email protected]>
Thanks but I did not suggest this.
What I have in v2 is RCU based. I hope Oliver or someone else can help make that work. Otherwise we can just drop this for now and revisit later.
(I have no problems with this patch if the Arm folks think the RCU-based version doesn't have a good ROI.)
Re: [PATCH v3 6/7] KVM: arm64: Participate in bitmap-based PTE aging
Quoted 4 lines · Yu Zhao
Quoted 1 line · James Houghton
Suggested-by Yu Zhao <[email protected]>Thanks but I did not suggest this.
Entirely up to you, but I would still want to credit everyone who contributed to a feature even if the underlying implementation has changed since the original attempt.
Quoted 2 lines
What I have in v2 is RCU based. I hope Oliver or someone else can help make that work.
Why? If there's data to show that RCU has a material benefit over taking the MMU lock for read then I'm all for it. Otherwise, the work required to support page-table walkers entirely outside of the MMU lock isn't justified.
In addition to ensuring that page table teardown is always RCU-safe, we'd need to make sure all of the walkers that take the write lock are prepared to handle races.
Quoted 2 lines
Otherwise we can just drop this for now and revisit later.
I really wouldn't get hung up on the locking as the make-or-break for whether arm64 supports this MMU notifier.
Signature · 2 lines
Thanks, Oliver
Marc Zyngier
maintainerRe: [PATCH v3 6/7] KVM: arm64: Participate in bitmap-based PTE aging
In reply to Yu ZhaoQuoted 9 lines · Yu Zhao
Quoted 4 lines · James Houghton
Participate in bitmap-based aging while grabbing the KVM MMU lock for reading. Ideally we wouldn't need to grab this lock at all, but that would require a more intrustive and risky change.
^^^^^^^^^^ intrusiveThis sounds subjective -- I'd just present the challenges and let reviewers make their own judgements.
Quite the opposite.
This sort of comment actually indicates that the author has at least understood some of the complexity behind the proposed changes. It is a qualitative comment that conveys useful information to reviewers, and even more to the maintainers of this code.
That's the difference between a human developer and a bot, and I'm not overly fond of bots.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
PATCH v3 7/7 mm: multi-gen LRU: use mmu_notifier_test_clear_young()
In reply to James HoughtonFrom: Yu Zhao <[email protected]>
Use mmu_notifier_{test,clear}_young_bitmap() to handle KVM PTEs in batches when the fast path is supported. This reduces the contention on kvm->mmu_lock when the host is under heavy memory pressure.
An existing selftest can quickly demonstrate the effectiveness of this patch. On a generic workstation equipped with 128 CPUs and 256GB DRAM:
$ sudo max_guest_memory_test -c 64 -m 250 -s 250
MGLRU run2
------------------
Before [1] ~64s
After ~51s
kswapd (MGLRU before)
100.00% balance_pgdat
100.00% shrink_node
100.00% shrink_one
99.99% try_to_shrink_lruvec
99.71% evict_folios
97.29% shrink_folio_list
==>> 13.05% folio_referenced
12.83% rmap_walk_file
12.31% folio_referenced_one
7.90% __mmu_notifier_clear_young
7.72% kvm_mmu_notifier_clear_young
7.34% _raw_write_lockkswapd (MGLRU after)
100.00% balance_pgdat
100.00% shrink_node
100.00% shrink_one
99.99% try_to_shrink_lruvec
99.59% evict_folios
80.37% shrink_folio_list
==>> 3.74% folio_referenced
3.59% rmap_walk_file
3.19% folio_referenced_one
2.53% lru_gen_look_around
1.06% __mmu_notifier_test_clear_young[1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to apples.
https://lore.kernel.org/r/[email protected]/
--- Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 183 ++++++++++++++---- 4 files changed, 159 insertions(+), 45 deletions(-)
Documentation/admin-guide/mm/multigen_lru.rst +5 -1
@@ -48,6 +48,10 @@ Values Components verified on x86 varieties other than Intel and AMD. If it is disabled, the multi-gen LRU will suffer a negligible performance degradation. +0x0008 Clearing the accessed bit in KVM page table entries in large + batches, when KVM MMU sets it (e.g., on x86_64). This can + improve the performance of guests when the host is under memory + pressure. [yYnN] Apply to all the components above. ====== ===============================================================
@@ -56,7 +60,7 @@ E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled - 0x0007 + 0x000f echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005
include/linux/mmzone.h +4 -2
@@ -397,6 +397,7 @@ enum {LRU_GEN_CORE ,LRU_GEN_MM_WALK ,LRU_GEN_NONLEAF_YOUNG , +LRU_GEN_KVM_MMU_WALK , NR_LRU_GEN_CAPS };
@@ -554,7 +555,7 @@ struct lru_gen_memcg {void lru_gen_init_pgdat (struct pglist_data *pgdat);void lru_gen_init_lruvec (struct lruvec *lruvec); -void lru_gen_look_around (struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw);void lru_gen_init_memcg (struct mem_cgroup *memcg);void lru_gen_exit_memcg (struct mem_cgroup *memcg);
@@ -573,8 +574,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } -static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw) { +return false; }static inline void lru_gen_init_memcg (struct mem_cgroup *memcg)
mm/rmap.c +3 -6
@@ -871,13 +871,10 @@ static bool folio_referenced_one(struct folio *folio,continue ; } -if (pvmw.pte) { -if (lru_gen_enabled () && -pte_young (ptep_get (pvmw.pte))) { -lru_gen_look_around (&pvmw); +if (lru_gen_enabled () && pvmw.pte) { +if (lru_gen_look_around (&pvmw)) referenced++; - } - + }else if (pvmw.pte) {if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) referenced++;
mm/vmscan.c +147 -36
362 lines not on this page · raw message
Signature · 1 line
2.44.0.478.gd926399ef9-goog
Quoted 5 lines · James Houghton
This patchset adds a fast path in KVM to test and clear access bits on sptes without taking the
mmu_lock. It also adds support for using a bitmap to (1) test the access bits for many sptes in a single call tommu_notifier_test_young, and to (2) clear the access bits for many ptes in a single call tommu_notifier_clear_young.
How much improvement would we get if we _just_ made test/clear_young lockless on x86 and hold the read-lock on arm64? And then how much benefit does the bitmap look-around add on top of that?
Quoted 11 lines · David Matlack
Quoted 5 lines · James Houghton
This patchset adds a fast path in KVM to test and clear access bits on sptes without taking the
mmu_lock. It also adds support for using a bitmap to (1) test the access bits for many sptes in a single call tommu_notifier_test_young, and to (2) clear the access bits for many ptes in a single call tommu_notifier_clear_young.How much improvement would we get if we _just_ made test/clear_young lockless on x86 and hold the read-lock on arm64? And then how much benefit does the bitmap look-around add on top of that?
I don't have these results right now. For the next version I will (1) separate the series into the locking change and the bitmap change, and I will (2) have performance data for each change separately. It is conceivable that the bitmap change should just be considered as a completely separate patchset.
Quoted 9 lines · James Houghton
Quoted 8 lines · David Matlack
Quoted 5 lines · James Houghton
This patchset adds a fast path in KVM to test and clear access bits on sptes without taking the
mmu_lock. It also adds support for using a bitmap to (1) test the access bits for many sptes in a single call tommu_notifier_test_young, and to (2) clear the access bits for many ptes in a single call tommu_notifier_clear_young.
Quoted 3 lines
Quoted 3 lines
How much improvement would we get if we _just_ made test/clear_young lockless on x86 and hold the read-lock on arm64? And then how much benefit does the bitmap look-around add on top of that?
Thanks David for providing the suggestion.
Quoted 5 lines
I don't have these results right now. For the next version I will (1) separate the series into the locking change and the bitmap change, and I will (2) have performance data for each change separately. It is conceivable that the bitmap change should just be considered as a completely separate patchset.
That'd be great. Having the performance numbers will make it even more compelling, but I'd be tempted to go for the lock improvement just because it doesn't add any new complexity and leverages existing patterns in the architectures that people seem to want improvements for.
The bitmap interface, OTOH, is rather complex. At least the current implementation breaks some of the isolation we have between the MMU code and the page table walker library on arm64, which I'm not ecstatic about. It _could_ be justified by a massive performance uplift over locking, but it'd have to be a sizable win.
Signature · 2 lines
Thanks, Oliver