KVM: arm64: Skip RCU protection for hyp stage-1
cover KVM: arm64: Skip RCU protection for hyp stage-1
Whelp, that was quick.
Marek reports [1] that the parallel faults series leads to a kernel BUG when initializing the hyp stage-1 page tables. Work around the issue by never acquiring the RCU read lock when walking hyp stage-1. This is safe because hyp stage-1 is protected by a spinlock (pKVM) or mutex (regular nVHE).
The included patch applies to the parallel faults series. To avoid breaking bisection, the patch should immediately precede commit c3119ae45dfb ("KVM: arm64: Protect stage-2 traversal with RCU"). Or, if preferred, I can respin the whole series in the correct order.
Tested with the pKVM isolated vCPU state series [2] merged on top, w/ kvm-arm.mode={nvhe,protected} on an Ampere Altra system.
[1]: https://lore.kernel.org/kvmarm/[email protected]/
[2]: https://lore.kernel.org/linux-arm-kernel/[email protected]/
Oliver Upton (1):
KVM: arm64: Use a separate function for hyp stage-1 walks
arch/arm64/include/asm/kvm_pgtable.h | 24 ++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 18 +++++++++++++++--- 3 files changed, 40 insertions(+), 4 deletions(-)
Signature · 1 line
2.38.1.431.g37b22c650d-goog
PATCH 1/1 KVM: arm64: Use a separate function for hyp stage-1 walks
A subsequent change to the page table walkers adds RCU protection for walking stage-2 page tables. KVM uses a global lock to serialize hyp stage-1 walks, meaning RCU protection is quite meaningless for protecting hyp stage-1 walkers.
Add a new helper, kvm_pgtable_hyp_walk(), for use when walking hyp stage-1 tables. Call directly into __kvm_pgtable_walk() as table concatenation is not a supported feature at stage-1.
No functional change intended.
--- arch/arm64/include/asm/kvm_pgtable.h | 24 ++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 18 +++++++++++++++--- 3 files changed, 40 insertions(+), 4 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +24 -0
@@ -596,6 +596,30 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);int kvm_pgtable_walk (struct kvm_pgtable *pgt,u64 addr,u64 size,struct kvm_pgtable_walker *walker); +/** +* kvm_pgtable_hyp_walk() - Walk a hyp stage-1 page-table. +* @pgt: Page-table structure initialized by kvm_pgtable_hyp_init(). +* @addr: Input address for the start of the walk. +* @size: Size of the range to walk. +* @walker: Walker callback description. +* +* The offset of @addr within a page is ignored and @size is rounded-up to +* the next page boundary. +* +* The walker will walk the page-table entries corresponding to the input +* address range specified, visiting entries according to the walker flags. +* Invalid entries are treated as leaf entries. Leaf entries are reloaded +* after invoking the walker callback, allowing the walker to descend into +* a newly installed table. +* +* Returning a negative error code from the walker callback function will +* terminate the walk immediately with the same error code. +* +* Return: 0 on success, negative error code on failure. +*/ +int kvm_pgtable_hyp_walk (struct kvm_pgtable *pgt,u64 addr,u64 size, +struct kvm_pgtable_walker *walker); +/** * kvm_pgtable_get_leaf() - Walk a page-table and retrieve the leaf entry * with its level.
arch/arm64/kvm/hyp/nvhe/setup.c +1 -1
@@ -246,7 +246,7 @@ static int finalize_host_mappings(void)struct memblock_region *reg = &hyp_memory[i];u64 start = (u64 )hyp_phys_to_virt (reg->base); - ret =kvm_pgtable_walk (&pkvm_pgtable, start, reg->size, &walker); + ret =kvm_pgtable_hyp_walk (&pkvm_pgtable, start, reg->size, &walker);if (ret)return ret; }
arch/arm64/kvm/hyp/pgtable.c +15 -3
@@ -335,6 +335,18 @@ int kvm_pgtable_get_leaf(struct kvm_pgtable *pgt, u64 addr,return ret; } +int kvm_pgtable_hyp_walk (struct kvm_pgtable *pgt,u64 addr,u64 size, +struct kvm_pgtable_walker *walker) +{ +struct kvm_pgtable_walk_data data = { + .walker = walker, + .addr =ALIGN_DOWN (addr,PAGE_SIZE ), + .end =PAGE_ALIGN (addr + size), + }; + +return __kvm_pgtable_walk (&data, pgt->mm_ops, pgt->pgd, pgt->start_level); +} +struct hyp_map_data {u64 phys;kvm_pte_t attr;
@@ -454,7 +466,7 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys,if (ret)return ret; - ret =kvm_pgtable_walk (pgt, addr, size, &walker); + ret =kvm_pgtable_hyp_walk (pgt, addr, size, &walker);dsb (ishst);isb ();return ret;
@@ -512,7 +524,7 @@ u64 kvm_pgtable_hyp_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)if (!pgt->mm_ops->page_count)return 0 ; -kvm_pgtable_walk (pgt, addr, size, &walker); +kvm_pgtable_hyp_walk (pgt, addr, size, &walker);return unmapped; }
@@ -557,7 +569,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; -WARN_ON (kvm_pgtable_walk (pgt,0 ,BIT (pgt->ia_bits), &walker)); +WARN_ON (kvm_pgtable_hyp_walk (pgt,0 ,BIT (pgt->ia_bits), &walker)); pgt->mm_ops->put_page (kvm_dereference_pteref (pgt->pgd, false)); pgt->pgd =NULL ; }
Signature · 1 line
2.38.1.431.g37b22c650d-goog
Will Deacon
maintainerRe: [PATCH 1/1] KVM: arm64: Use a separate function for hyp stage-1 walks
Quoted 39 lines · Oliver Upton
A subsequent change to the page table walkers adds RCU protection for walking stage-2 page tables. KVM uses a global lock to serialize hyp stage-1 walks, meaning RCU protection is quite meaningless for protecting hyp stage-1 walkers.
Add a new helper,
kvm_pgtable_hyp_walk(), for use when walking hyp stage-1 tables. Call directly into__kvm_pgtable_walk()as table concatenation is not a supported feature at stage-1.No functional change intended.
Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/kvm_pgtable.h | 24 ++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 18 +++++++++++++++--- 3 files changed, 40 insertions(+), 4 deletions(-)arch/arm64/include/asm/kvm_pgtable.h +13
-0@@ -596,6 +596,30 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);int kvm_pgtable_walk (struct kvm_pgtable *pgt,u64 addr,u64 size,struct kvm_pgtable_walker *walker); +/** +* kvm_pgtable_hyp_walk() - Walk a hyp stage-1 page-table. +* @pgt: Page-table structure initialized by kvm_pgtable_hyp_init(). +* @addr: Input address for the start of the walk. +* @size: Size of the range to walk. +* @walker: Walker callback description. +* +* The offset of @addr within a page is ignored and @size is rounded-up to +* the next page boundary. +* +* The walker will walk the page-table entries corresponding to the input +* address range specified, visiting entries according to the walker flags. +* Invalid entries are treated as leaf entries. Leaf entries are reloaded
arch/arm64/include/asm/kvm_pgtable.h int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); · line 621
Quoted 10 lines
arch/arm64/include/asm/kvm_pgtable.h +10
-0+* after invoking the walker callback, allowing the walker to descend into +* a newly installed table. +* +* Returning a negative error code from the walker callback function will +* terminate the walk immediately with the same error code. +* +* Return: 0 on success, negative error code on failure. +*/ +int kvm_pgtable_hyp_walk (struct kvm_pgtable *pgt,u64 addr,u64 size, +struct kvm_pgtable_walker *walker);
Hmm, this feels like slightly the wrong abstraction to me -- there's nothing hyp-specific about the problem being solved, it's just that the only user is for hyp walks.
Could we instead rework 'struct kvm_pgtable' slightly so that the existing 'flags' field is no-longer stage-2 specific and includes a KVM_PGTABLE_LOCKED flag which could be set by kvm_pgtable_hyp_init()?
That way the top-level API remains unchanged and the existing callers will continue to work.
Cheers,
Will
Re: [PATCH 1/1] KVM: arm64: Use a separate function for hyp stage-1 walks
Hey Will,
On Tue, Nov 15, 2022 at 01:25:34PM +0000, Will Deacon wrote:
[...]
Quoted 8 lines · Oliver Upton
arch/arm64/include/asm/kvm_pgtable.h int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); · line 621
Quoted 2 lines
arch/arm64/include/asm/kvm_pgtable.h +2
-0+int kvm_pgtable_hyp_walk (struct kvm_pgtable *pgt,u64 addr,u64 size, +struct kvm_pgtable_walker *walker);Hmm, this feels like slightly the wrong abstraction to me -- there's nothing hyp-specific about the problem being solved, it's just that the only user is for hyp walks.
Quoted 6 lines
Could we instead rework 'struct
kvm_pgtable' slightly so that the existing 'flags' field is no-longer stage-2 specific and includes aKVM_PGTABLE_LOCKEDflag which could be set bykvm_pgtable_hyp_init()?That way the top-level API remains unchanged and the existing callers will continue to work.
Thanks for the suggestion! Yeah, this should be described by the flags instead.
We already have KVM_PGTABLE_WALK_SHARED, I could actually condition the RCU lock/unlock on that one. That would make it an explicit opt-in instead of requiring an opt out with callers passing KVM_PGTABLE_LOCKED.
Thoughts?
--
Thanks,
Oliver