xen/riscv: introduce p2m functionality
cover xen/riscv: introduce p2m functionality
In this patch series are introduced necessary functions to build and manage RISC-V guest page tables and MMIO/RAM mappings.
This patch series is based on the patch [1]:
https://lore.kernel.org/xen-devel/da9273c20dc7ac1c131322e38a8cef361dfd86a9.1746530883.git.oleksii.kurochko@gmail.com/T/#u
Oleksii Kurochko (6):
xen/riscv: add inclusion of xen/bitops.h to asm/cmpxchg.h xen/riscv: introduce things necessary for p2m initialization xen/riscv: construct the P2M pages pool for guests xen/riscv: define pt_t and pt_walk_t structures
xen/riscv: add new p2m types and helper macros for type classification xen/riscv: implement p2m mapping functionality
xen/arch/riscv/Makefile | 1 + xen/arch/riscv/include/asm/cmpxchg.h | 1 + xen/arch/riscv/include/asm/domain.h | 16 + xen/arch/riscv/include/asm/mm.h | 36 +- xen/arch/riscv/include/asm/p2m.h | 121 ++- xen/arch/riscv/include/asm/page.h | 65 +- xen/arch/riscv/p2m.c | 1015 ++++++++++++++++++++++++++ 7 files changed, 1243 insertions(+), 12 deletions(-) create mode 100644 xen/arch/riscv/p2m.c
Signature · 1 line
2.49.0
PATCH 1/6 xen/riscv: add inclusion of xen/bitops.h to asm/cmpxchg.h
Add inclusion of xen/bitops.h to asm/cmpxchg.h to avoid compilation issues connected to GENMASK() which is used inside asm/cmpxchg.h.
---
This patch should go first; otherwise one of the further patches of this patch series could face a compilation issue.
--- xen/arch/riscv/include/asm/cmpxchg.h | 1 + 1 file changed, 1 insertion(+)
xen/arch/riscv/include/asm/cmpxchg.h +1 -0
@@ -4,6 +4,7 @@#ifndef ASM__RISCV__CMPXCHG_H #define ASM__RISCV__CMPXCHG_H +#include <xen/bitops.h> #include <xen/compiler.h> #include <xen/lib.h>
Signature · 1 line
2.49.0
Re: [PATCH v1 1/6] xen/riscv: add inclusion of xen/bitops.h to asm/cmpxchg.h
Quoted 4 lines · Oleksii Kurochko
Add inclusion of
xen/bitops.htoasm/cmpxchg.hto avoid compilation issues connected toGENMASK()which is used insideasm/cmpxchg.h.Signed-off-by Oleksii Kurochko <[email protected]>
PATCH 2/6 xen/riscv: introduce things necessary for p2m initialization
In reply to Oleksii KurochkoIntroduce the following things:
- p2m_domain structure which describe per p2m-table state.
- Update arch_domain structure with the mentioned above structure.
- p2m_get_hostp2m() to recieve domain's p2m structure.
- Introudce p2m_write_lock() and p2m_is_write_locked().
- p2m_init() to initalize p2m:
- allocate p2m table by using of p2m_alloc_table().
- initialize lock premitive necessary to protect updates to the p2m.
- Introduce the following functions to implement p2m_alloc_table():
- p2m_allocate_root() to allocate p2m root table by using another introduced helpers p2m_get_clean_page() and clear_and_clean_page().
- introduce p2m_force_tlb_flush_sync() to flush TLBs after p2m table allocation before being used. (it isn't necessary at the current stage of development but could be useful once the VMID is marked unused, a new domain can reuse the VMID for its own. If the TLB is not flushed, entries can contain wrong translation.)
- Implement maddr_to_page() and page_to_maddr().
--- xen/arch/riscv/Makefile | 1 + xen/arch/riscv/include/asm/domain.h | 6 + xen/arch/riscv/include/asm/mm.h | 4 + xen/arch/riscv/include/asm/p2m.h | 76 +++++++++++++ xen/arch/riscv/p2m.c | 168 ++++++++++++++++++++++++++++ 5 files changed, 255 insertions(+) create mode 100644 xen/arch/riscv/p2m.c
xen/arch/riscv/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-y += entry.o obj-y += intc.o obj-y += mm.o obj-y += pt.o +obj-y += p2m.o obj-$(CONFIG_RISCV_64) += riscv64/ obj-y += sbi.o obj-y += setup.o
xen/arch/riscv/include/asm/domain.h +6 -0
@@ -5,6 +5,8 @@#include <xen/xmalloc.h> #include <public/hvm/params.h> +#include <asm/p2m.h> +struct hvm_domain {uint64_t params[HVM_NR_PARAMS ];
@@ -16,8 +18,12 @@ struct arch_vcpu_io {struct arch_vcpu { }; +struct arch_domain {struct hvm_domain hvm; + +struct p2m_domain p2m; + };#include <xen/sched.h>
xen/arch/riscv/include/asm/mm.h +4 -0
@@ -149,6 +149,10 @@ extern struct page_info *frametable_virt_start;#define mfn_to_page (mfn) (frametable_virt_start + mfn_x(mfn))#define page_to_mfn (pg) _mfn((pg) - frametable_virt_start) +/* Convert between machine addresses and page-info structures. */ +#define maddr_to_page (ma) mfn_to_page(maddr_to_mfn(ma)) +#define page_to_maddr (pg) (mfn_to_maddr(page_to_mfn(pg))) +static inline void *page_to_virt (const struct page_info *pg) {return mfn_to_virt (mfn_x (page_to_mfn (pg)));
xen/arch/riscv/include/asm/p2m.h +76 -0
93 lines not on this page · raw message
xen/arch/riscv/p2m.c new file +168 -0
168 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
xen/arch/riscv/include/asm/p2m.h line 10
Quoted 13 lines · Oleksii Kurochko
xen/arch/riscv/include/asm/p2m.h +5
-0@@ -3,11 +3,73 @@#define ASM__RISCV__P2M_H #include <xen/errno.h> +#include <xen/mem_access.h> +#include <xen/mm.h> +#include <xen/radix-tree.h> +#include <xen/rwlock.h> +#include <xen/types.h>
We're phasing out the inclusion of xen/types.h for complex headers like this, as it's pulled in by almost all dependencies.
Quoted 29 lines
xen/arch/riscv/p2m.c new file +23
-0@@ -0,0 +1,168 @@ +#include <xen/domain_page.h> +#include <xen/iommu.h> +#include <xen/lib.h> +#include <xen/mm.h> +#include <xen/pfn.h> +#include <xen/rwlock.h> +#include <xen/sched.h> +#include <xen/spinlock.h> + +#include <asm/page.h> +#include <asm/p2m.h> + +/* +* Force a synchronous P2M TLB flush. +* +* Must be called with the p2m lock held. +* +* TODO: add support of flushing TLB connected to VMID. +*/ +static void p2m_force_tlb_flush_sync (struct p2m_domain *p2m) +{ +ASSERT (p2m_is_write_locked (p2m)); +
xen/arch/riscv/p2m.c line 33
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+/* +* TODO: shouldn't be this flush done for each physical CPU? +* If yes, then SBI call sbi_remote_hfence_gvma() could +* be used for that. +*/ +#if defined(__riscv_hh) || defined(__riscv_h) + asmvolatile ("hfence.gvma" :::"memory" ); +#else + asmvolatile (".insn r 0x73, 0x0, 0x31, x0, x0, x0" :::"memory" ); +#endif
TLB flushing needs to happen for each pCPU which potentially has cached a mapping.
In other arches, this is tracked by d->dirty_cpumask which is the bitmap of pCPUs where this domain is scheduled.
CPUs need to flush their TLBs before removing themselves from d->dirty_cpumask, which is typically done during context switch, but it means that to flush the P2M, you only need to IPI a subset of CPUs.
Quoted 36 lines
xen/arch/riscv/p2m.c new file +36
-0+ + p2m->need_flush = false; +} + +static void p2m_tlb_flush_sync (struct p2m_domain *p2m) +{ +if ( p2m->need_flush ) +p2m_force_tlb_flush_sync (p2m); +} + +/* Unlock the flush and do a P2M TLB flush if necessary */ +void p2m_write_unlock (struct p2m_domain *p2m) +{ +/* +* The final flush is done with the P2M write lock taken to avoid +* someone else modifying the P2M wbefore the TLB invalidation has +* completed. +*/ +p2m_tlb_flush_sync (p2m); + +write_unlock (&p2m->lock); +} + +static void clear_and_clean_page (struct page_info *page) +{ +void *p =__map_domain_page (page); + +clear_page (p); +unmap_domain_page (p); +} + +static struct page_info *p2m_get_clean_page (struct domain *d) +{ +struct page_info *page; + +/*
xen/arch/riscv/p2m.c line 79
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + page =alloc_domheap_pages (NULL ,2 ,0 ); +if ( page ==NULL ) +return NULL ; + +clear_and_clean_page (page);
You appear to have allocated 4 pages, but only zeroed one.
~Andrew
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 43 lines · Andrew Cooper
Quoted 29 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c new file +23
-0@@ -0,0 +1,168 @@ +#include <xen/domain_page.h> +#include <xen/iommu.h> +#include <xen/lib.h> +#include <xen/mm.h> +#include <xen/pfn.h> +#include <xen/rwlock.h> +#include <xen/sched.h> +#include <xen/spinlock.h> + +#include <asm/page.h> +#include <asm/p2m.h> + +/* +* Force a synchronous P2M TLB flush. +* +* Must be called with the p2m lock held. +* +* TODO: add support of flushing TLB connected to VMID. +*/ +static void p2m_force_tlb_flush_sync (struct p2m_domain *p2m) +{ +ASSERT (p2m_is_write_locked (p2m)); +xen/arch/riscv/p2m.c line 33
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+/* +* TODO: shouldn't be this flush done for each physical CPU? +* If yes, then SBI call sbi_remote_hfence_gvma() could +* be used for that. +*/ +#if defined(__riscv_hh) || defined(__riscv_h) + asmvolatile ("hfence.gvma" :::"memory" ); +#else + asmvolatile (".insn r 0x73, 0x0, 0x31, x0, x0, x0" :::"memory" ); +#endif TLB flushing needs to happen for each pCPU which potentially has cached a mapping.
Quoted 2 lines
In other arches, this is tracked by
d->dirty_cpumaskwhich is the bitmap of pCPUs where this domain is scheduled.
I could only find usage of|d->dirty_cpumask| in x86 and common code (grant tables) for flushing the TLB. However, it seems that|d->dirty_cpumask| is not set anywhere for ARM. Is it sufficient to set a bit in|d->dirty_cpumask| in|startup_cpu_idle_loop()|?
In addition, it’s also necessary to set and clear bits in|d->dirty_cpumask| during|context_switch|, correct? Set it before switching from the previous domain, and clear it after switching to the new domain?
Also, when a bit is set in|d->dirty_cpumask|, the|v->processor| value is also stored in|v->dirty_cpu|. Is this needed to track which processor is currently being used for the vCPU?
Quoted 3 lines
CPUs need to flush their TLBs before removing themselves from
d->dirty_cpumask, which is typically done during context switch, but it means that to flush the P2M, you only need to IPI a subset of CPUs.
I can't find where the P2M flush happens for x86/ARM. Could you please point me to where it is handled?
Also, I found guest_flush_tlb_mask() for x86. I assume that it is x86 specific and generally it is enough to have only flush_tlb_mask(), right?
Thanks in advance for the answers.
~ Oleksii
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 46 lines · Oleksii Kurochko
Quoted 43 lines · Andrew Cooper
Quoted 29 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c new file +23
-0@@ -0,0 +1,168 @@ +#include <xen/domain_page.h> +#include <xen/iommu.h> +#include <xen/lib.h> +#include <xen/mm.h> +#include <xen/pfn.h> +#include <xen/rwlock.h> +#include <xen/sched.h> +#include <xen/spinlock.h> + +#include <asm/page.h> +#include <asm/p2m.h> + +/* +* Force a synchronous P2M TLB flush. +* +* Must be called with the p2m lock held. +* +* TODO: add support of flushing TLB connected to VMID. +*/ +static void p2m_force_tlb_flush_sync (struct p2m_domain *p2m) +{ +ASSERT (p2m_is_write_locked (p2m)); +xen/arch/riscv/p2m.c line 33
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+/* +* TODO: shouldn't be this flush done for each physical CPU? +* If yes, then SBI call sbi_remote_hfence_gvma() could +* be used for that. +*/ +#if defined(__riscv_hh) || defined(__riscv_h) + asmvolatile ("hfence.gvma" :::"memory" ); +#else + asmvolatile (".insn r 0x73, 0x0, 0x31, x0, x0, x0" :::"memory" ); +#endif TLB flushing needs to happen for each pCPU which potentially has cached a mapping.
Quoted 6 lines
Quoted 2 lines
In other arches, this is tracked by
d->dirty_cpumaskwhich is the bitmap of pCPUs where this domain is scheduled.I could only find usage of|d->dirty_cpumask| in x86 and common code (grant tables) for flushing the TLB. However, it seems that|d->dirty_cpumask| is not set anywhere for ARM. Is it sufficient to set a bit in|d->dirty_cpumask| in|startup_cpu_idle_loop()|?
And one more thing.
If|d->dirty_cpumask| is empty (for example, on p2m initialization stage) then p2m TLB flush could be skipped at all, right?
~ Oleksii
Quoted 17 lines
In addition, it’s also necessary to set and clear bits in|d->dirty_cpumask| during|context_switch|, correct? Set it before switching from the previous domain, and clear it after switching to the new domain?
Also, when a bit is set in|d->dirty_cpumask|, the|v->processor| value is also stored in|v->dirty_cpu|. Is this needed to track which processor is currently being used for the vCPU?Quoted 3 lines
CPUs need to flush their TLBs before removing themselves from
d->dirty_cpumask, which is typically done during context switch, but it means that to flush the P2M, you only need to IPI a subset of CPUs.I can't find where the P2M flush happens for x86/ARM. Could you please point me to where it is handled?
Also, I found
guest_flush_tlb_mask()for x86. I assume that it is x86 specific and generally it is enough to have onlyflush_tlb_mask(), right?Thanks in advance for the answers.
~ Oleksii
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
In reply to Oleksii KurochkoQuoted 41 lines · Oleksii Kurochko
Quoted 40 lines · Andrew Cooper
Quoted 26 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c new file +23
-0@@ -0,0 +1,168 @@ +#include <xen/domain_page.h> +#include <xen/iommu.h> +#include <xen/lib.h> +#include <xen/mm.h> +#include <xen/pfn.h> +#include <xen/rwlock.h> +#include <xen/sched.h> +#include <xen/spinlock.h> + +#include <asm/page.h> +#include <asm/p2m.h> + +/* +* Force a synchronous P2M TLB flush. +* +* Must be called with the p2m lock held. +* +* TODO: add support of flushing TLB connected to VMID. +*/ +static void p2m_force_tlb_flush_sync (struct p2m_domain *p2m) +{ +ASSERT (p2m_is_write_locked (p2m)); +xen/arch/riscv/p2m.c line 33
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+/* +* TODO: shouldn't be this flush done for each physical CPU? +* If yes, then SBI call sbi_remote_hfence_gvma() could +* be used for that. +*/ +#if defined(__riscv_hh) || defined(__riscv_h) + asmvolatile ("hfence.gvma" :::"memory" ); +#else + asmvolatile (".insn r 0x73, 0x0, 0x31, x0, x0, x0" :::"memory" ); +#endif TLB flushing needs to happen for each pCPU which potentially has cached a mapping.
Quoted 7 lines
Quoted 2 lines
In other arches, this is tracked by
d->dirty_cpumaskwhich is the bitmap of pCPUs where this domain is scheduled.I could only find usage of|d->dirty_cpumask| in x86 and common code (grant tables) for flushing the TLB. However, it seems that|d->dirty_cpumask| is not set anywhere for ARM. Is it sufficient to set a bit in|d->dirty_cpumask| in|startup_cpu_idle_loop()|?
No, how would the idle loop be relevant here? The bit needs setting for any pCPU a vCPU of the domain is running on, i.e. somewhere in context switch code.
Quoted 8 lines
In addition, it’s also necessary to set and clear bits in|d->dirty_cpumask| during|context_switch|, correct? Set it before switching from the previous domain, and clear it after switching to the new domain?
Also, when a bit is set in|d->dirty_cpumask|, the|v->processor| value is also stored in|v->dirty_cpu|. Is this needed to track which processor is currently being used for the vCPU?
Quoted 6 lines
Quoted 3 lines
CPUs need to flush their TLBs before removing themselves from
d->dirty_cpumask, which is typically done during context switch, but it means that to flush the P2M, you only need to IPI a subset of CPUs.I can't find where the P2M flush happens for x86/ARM. Could you please point me to where it is handled?
Grep for ept_sync_domain, which will give you several involved functions (for the Intel, i.e. EPT case).
Quoted 2 lines
Also, I found
guest_flush_tlb_mask()for x86. I assume that it is x86 specific and generally it is enough to have onlyflush_tlb_mask(), right?
Yes, that's an x86-specific helper which you may or may not want a counterpart of.
Jan
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
In reply to Oleksii KurochkoQuoted 12 lines · Oleksii Kurochko
xen/arch/riscv/include/asm/domain.h +2
-0@@ -5,6 +5,8 @@#include <xen/xmalloc.h> #include <public/hvm/params.h> +#include <asm/p2m.h> +struct hvm_domain {uint64_t params[HVM_NR_PARAMS ];
xen/arch/riscv/include/asm/domain.h struct arch_vcpu_io { · line 21
Quoted 4 lines
xen/arch/riscv/include/asm/domain.h +1
-0@@ -16,8 +18,12 @@ struct arch_vcpu_io {struct arch_vcpu { }; +
Nit: As before, no double blank lines please.
xen/arch/riscv/include/asm/domain.h struct arch_vcpu_io { · line 27
Quoted 6 lines
xen/arch/riscv/include/asm/domain.h +3
-0struct arch_domain {struct hvm_domain hvm; + +struct p2m_domain p2m; + };
Similarly, no blank lines please at the end of a struct/union/enum.
xen/arch/riscv/include/asm/mm.h extern struct page_info *frametable_virt_start; · line 154
Quoted 9 lines
xen/arch/riscv/include/asm/mm.h +3
-0@@ -149,6 +149,10 @@ extern struct page_info *frametable_virt_start;#define mfn_to_page (mfn) (frametable_virt_start + mfn_x(mfn))#define page_to_mfn (pg) _mfn((pg) - frametable_virt_start) +/* Convert between machine addresses and page-info structures. */ +#define maddr_to_page (ma) mfn_to_page(maddr_to_mfn(ma)) +#define page_to_maddr (pg) (mfn_to_maddr(page_to_mfn(pg)))
Nit: The outermost parentheses aren't really needed here. Or if you really want them, then please be consistent and also add them for the other macro you add.
Quoted 22 lines
xen/arch/riscv/p2m.c new file +19
-0@@ -0,0 +1,168 @@ +#include <xen/domain_page.h> +#include <xen/iommu.h> +#include <xen/lib.h> +#include <xen/mm.h> +#include <xen/pfn.h> +#include <xen/rwlock.h> +#include <xen/sched.h> +#include <xen/spinlock.h> + +#include <asm/page.h> +#include <asm/p2m.h> + +/* +* Force a synchronous P2M TLB flush. +* +* Must be called with the p2m lock held. +* +* TODO: add support of flushing TLB connected to VMID. +*/
xen/arch/riscv/p2m.c line 29
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+static void p2m_force_tlb_flush_sync (struct p2m_domain *p2m) +{ +ASSERT (p2m_is_write_locked (p2m)); + +/* +* TODO: shouldn't be this flush done for each physical CPU? +* If yes, then SBI call sbi_remote_hfence_gvma() could +* be used for that. +*/ +#if defined(__riscv_hh) || defined(__riscv_h)
This is a compiler capability check (which would be applicable if you used a built-in function below).
xen/arch/riscv/p2m.c line 30
Quoted 1 line
xen/arch/riscv/p2m.c new file +1
-0+ asmvolatile ("hfence.gvma" :::"memory" );
Whereas here you use a feature in the assembler, at least for the GNU toolchain.
xen/arch/riscv/p2m.c line 63
Quoted 7 lines
xen/arch/riscv/p2m.c new file +7
-0+static void clear_and_clean_page (struct page_info *page) +{ +void *p =__map_domain_page (page); + +clear_page (p); +unmap_domain_page (p); +}
What's the "clean" about in the function name? The "clear" is referring to the clear_page() call afaict. Also aren't you largely open-coding clear_domain_page() here?
xen/arch/riscv/p2m.c line 75
Quoted 11 lines
xen/arch/riscv/p2m.c new file +11
-0+static struct page_info *p2m_get_clean_page (struct domain *d) +{ +struct page_info *page; + +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + page =alloc_domheap_pages (NULL ,2 ,0 );
Shouldn't this allocation come from the domain's P2M pool (which is yet to be introduced)? Also hard-coding 2 here as order effectively builds in an assumption that PAGE_SIZE will only ever be 4k. I think to wants properly calculating instead.
xen/arch/riscv/p2m.c line 82
Quoted 7 lines
xen/arch/riscv/p2m.c new file +7
-0+if ( page ==NULL ) +return NULL ; + +clear_and_clean_page (page); + +return page; +}
Contrary to the function name you obtained 4 pages here, which is suitable for ...
xen/arch/riscv/p2m.c line 87
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+static struct page_info *p2m_allocate_root (struct domain *d) +{ +return p2m_get_clean_page (d); +}
... this but - I expect - no anywhere else.
xen/arch/riscv/p2m.c line 89
Quoted 1 line
xen/arch/riscv/p2m.c new file +1
-0+static unsigned long hgatp_from_page_info (struct page_info *page_info)
Except for the struct name please drop _info; we don#t use such anywhere else.
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+{ +unsigned long ppn; +unsigned long hgatp_mode; +
xen/arch/riscv/p2m.c line 103
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ ppn =PFN_DOWN (page_to_maddr (page_info)) &HGATP_PPN ; + +/* ASID (VMID) not supported yet */ + +#if RV_STAGE1_MODE ==SATP_MODE_SV39 + hgatp_mode =HGATP_MODE_SV39X4 ; +#elif RV_STAGE1_MODE ==SATP_MODE_SV48 + hgatp_mode =HGATP_MODE_SV48X4 ; +#else +#error "add HGATP_MODE"
As before, please have the # of pre-processor directives in the first column.
xen/arch/riscv/p2m.c line 106
Quoted 3 lines
xen/arch/riscv/p2m.c new file +3
-0+#endif + +return ppn | (hgatp_mode <<HGATP_MODE_SHIFT );
Use MASK_INSR()?
Quoted 9 lines
xen/arch/riscv/p2m.c new file +9
-0+} + +static int p2m_alloc_table (struct domain *d) +{ +struct p2m_domain *p2m =p2m_get_hostp2m (d); + + p2m->root =p2m_allocate_root (d); +if ( !p2m->root ) +return -ENOMEM ;
xen/arch/riscv/p2m.c line 125
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ + p2m->hgatp =hgatp_from_page_info (p2m->root); + +/* +* Make sure that all TLBs corresponding to the new VMID are flushed +* before using it. +*/ +p2m_write_lock (p2m); +p2m_force_tlb_flush_sync (p2m); +p2m_write_unlock (p2m);
While Andrew directed you towards a better model in general, it won't be usable here then, as the guest didn't run on any pCPU(s) yet. Imo you want to do a single global flush e.g. when VMIDs wrap around. That'll be fewer global flushes than one per VM creation.
xen/arch/riscv/p2m.c line 139
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+int p2m_init (struct domain *d) +{ +struct p2m_domain *p2m =p2m_get_hostp2m (d); +int rc; + +rwlock_init (&p2m->lock); +INIT_PAGE_LIST_HEAD (&p2m->pages); + + p2m->max_mapped_gfn =_gfn (0 ); + p2m->lowest_mapped_gfn =_gfn (ULONG_MAX );
You don't ever read these two fields. Likely better to introduce them when they're actually needed. Same possibly for further things done in this function.
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+ p2m->default_access = p2m_access_rwx; + +radix_tree_init (&p2m->p2m_type); +
xen/arch/riscv/p2m.c line 154
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+#ifdef CONFIG_HAS_PASSTHROUGH +/* +* Some IOMMUs don't support coherent PT walk. When the p2m is +* shared with the CPU, Xen has to make sure that the PT changes have +* reached the memory +*/ + p2m->clean_pte =is_iommu_enabled (d) && + !iommu_has_feature (d,IOMMU_FEAT_COHERENT_WALK ); +#else + p2m->clean_pte = true;
When there's no IOMMU (in use), doesn't this want to be "false"?
xen/arch/riscv/p2m.c line 161
Quoted 7 lines
xen/arch/riscv/p2m.c new file +7
-0+#endif + +/* +* "Trivial" initialisation is now complete. Set the backpointer so +* p2m_teardown() and friends know to do something. +*/ + p2m->domain = d;
And where is that p2m_teardown(), to cross-check the comment against?
Jan
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 12 lines · Jan Beulich
xen/arch/riscv/p2m.c line 63
Quoted 9 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c new file +7
-0+static void clear_and_clean_page (struct page_info *page) +{ +void *p =__map_domain_page (page); + +clear_page (p); +unmap_domain_page (p); +}What's the "clean" about in the function name? The "clear" is referring to the
clear_page()call afaict.
Missed to add clean_dcache_va_range() between clear_page() and unmap_domain_page().
Quoted 2 lines
Also aren't you largely open-coding
clear_domain_page()here?
Yes, missed that it is almost the sane as clear_domain_page(), so we could re-write this function as:
static void clear_and_clean_page(struct page_info *page)
{
clean_dcache_va_range(page, PAGE_SIZE);
clear_domain_page(page_to_mfn(page));}
Quoted 5 lines
Quoted 5 lines
xen/arch/riscv/p2m.c new file +5
-0+static struct page_info *p2m_get_clean_page (struct domain *d) +{ +struct page_info *page; + +/*
Quoted 8 lines
xen/arch/riscv/p2m.c line 75
Quoted 6 lines
xen/arch/riscv/p2m.c new file +6
-0+* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + page =alloc_domheap_pages (NULL ,2 ,0 );Shouldn't this allocation come from the domain's P2M pool (which is yet to be introduced)?
First, I will drop p2m_get_clean_page() as it will be used only for p2m root page table allocation.
p2m_init() is called by domain_create() [->arch_domain_create()->p2m_init()] from create_domUs():
[https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L984].
When p2m_init() is called, p2m pool isn't ready and domain isn't created yet. Last one is also crucial for usage of p2m pool as p2m pool belongs to domain and thereby it is using alloc_domheap_page(d, ...) (Not NULL as for allocation of p2m root table above), so domain should be created first.
And only after domain_create() will created domain, p2m pool could be initialized during domain construction:
https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L756
and the size of p2m pool depends on the value from memory property of domain node in DT. (line 748, the link the same as above).
Also, if CONFIG_ARCH_PAGING_MEMPOOL=n, then p2m pool isn't used. But it isn't a case for RISC-V for the moment. Probably one day it would be useful if someone wanted to add support for MMU-less case. Something like Arm is doing now for R-cores.
Quoted 3 lines
Also hard-coding 2 here as order effectively builds
in an assumption thatPAGE_SIZEwill only ever be 4k. I think to wants properly calculating instead.
I haven't thought about that. I will update it with:
page = alloc_domheap_pages(NULL, get_order_from_bytes(KB(16)), 0);
Quoted 11 lines
xen/arch/riscv/p2m.c line 82
Quoted 8 lines
xen/arch/riscv/p2m.c new file +7
-0+if ( page ==NULL ) +return NULL ; + +clear_and_clean_page (page); + +return page; +}Contrary to the function name you obtained 4 pages here, which is suitable for ...
Quoted 5 lines
xen/arch/riscv/p2m.c line 87
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+static struct page_info *p2m_allocate_root (struct domain *d) +{ +return p2m_get_clean_page (d); +}... this but - I expect - no anywhere else.
Totally agree, as mentioned above this function is used only for p2m_allocate_root(). I will just open-code it in p2m_allocate_root().
Quoted 16 lines
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+{ +unsigned long ppn; +unsigned long hgatp_mode; +xen/arch/riscv/p2m.c line 103
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ ppn =PFN_DOWN (page_to_maddr (page_info)) &HGATP_PPN ; + +/* ASID (VMID) not supported yet */ + +#if RV_STAGE1_MODE ==SATP_MODE_SV39 + hgatp_mode =HGATP_MODE_SV39X4 ; +#elif RV_STAGE1_MODE ==SATP_MODE_SV48 + hgatp_mode =HGATP_MODE_SV48X4 ; +#else +#error "add HGATP_MODE"As before, please have the # of pre-processor directives in the first column.
Quoted 4 lines
xen/arch/riscv/p2m.c line 106
Quoted 3 lines
xen/arch/riscv/p2m.c new file +3
-0+#endif + +return ppn | (hgatp_mode <<HGATP_MODE_SHIFT );Use
MASK_INSR()?
Do you mean MASK_INSR(hgatp_mode, HGATP_MODE_MASK)? If yes, then I didn't get what is the point then?
Quoted 16 lines
Quoted 16 lines
xen/arch/riscv/p2m.c new file +15
-0+} + +static int p2m_alloc_table (struct domain *d) +{ +struct p2m_domain *p2m =p2m_get_hostp2m (d); + + p2m->root =p2m_allocate_root (d); +if ( !p2m->root ) +return -ENOMEM ; + + p2m->hgatp =hgatp_from_page_info (p2m->root); + +/* +* Make sure that all TLBs corresponding to the new VMID are flushed +* before using it.
Quoted 8 lines
xen/arch/riscv/p2m.c line 125
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+*/ +p2m_write_lock (p2m); +p2m_force_tlb_flush_sync (p2m); +p2m_write_unlock (p2m);While Andrew directed you towards a better model in general, it won't be usable here then, as the guest didn't run on any pCPU(s) yet. Imo you want to do a single global flush e.g. when VMIDs wrap around. That'll be fewer global flushes than one per VM creation.
I am not sure that I get a phrase 'VMIDs wrap around'.
I am going to implement, p2m_force_tlb_flush_sync() as:
static void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) {
...
sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0);
...
}
With such implementation if the guest didn't run on any pCPU(s) yet
then d->dirty_cpumask is empty, then sbi_remote_hfence_gvma() will do nothing
as hmask will be NULL (https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/riscv/sbi.c?ref_type=heads#L238).
I am not sure that it is a good idea as I can't find a guarantee in the spec
that TLB will be empty during boot time.
But if another VM is being created then we should flush stage2 before run a VM so, the new VM won't re-use something from the old VM.
Or in case of VMID if VMID is reused by new VM in case if, for example, the previous owner(domain) was destroyed and a new domain is reusing VMID, it is needed to flush stage2.
p2m_alloc_table() looks a good place for that and I am not sure that we can do a single global flush, and I don't really know in first glance where it should be done.
Quoted 7 lines
Quoted 7 lines
xen/arch/riscv/p2m.c new file +7
-0+ p2m->default_access = p2m_access_rwx; + +radix_tree_init (&p2m->p2m_type); + +#ifdef CONFIG_HAS_PASSTHROUGH +/* +* Some IOMMUs don't support coherent PT walk. When the p2m is
Quoted 8 lines
xen/arch/riscv/p2m.c line 154
Quoted 7 lines
xen/arch/riscv/p2m.c new file +7
-0+* shared with the CPU, Xen has to make sure that the PT changes have +* reached the memory +*/ + p2m->clean_pte =is_iommu_enabled (d) && + !iommu_has_feature (d,IOMMU_FEAT_COHERENT_WALK ); +#else + p2m->clean_pte = true;When there's no IOMMU (in use), doesn't this want to be "false"?
I think you are right, "false" is more correct here.
Quoted 9 lines
xen/arch/riscv/p2m.c line 161
Quoted 8 lines
xen/arch/riscv/p2m.c new file +7
-0+#endif + +/* +* "Trivial" initialisation is now complete. Set the backpointer so +* p2m_teardown() and friends know to do something. +*/ + p2m->domain = d;And where is that
p2m_teardown(), to cross-check the comment against?
It is not introduced now as I expected it is need only when domain is needed to be stop for some reason. And it isn't really needed now.
Anyway, it seems like it is a stale comment as on other arch-es p2m_teardown() has an argument with struct domain *d.
I can update the commit to:
"Trivial" initialisation is now complete. Set the backpointer so the users of p2m
could get an access to domain structure.
~ Oleksii
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 19 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich
xen/arch/riscv/p2m.c line 75
Quoted 11 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c new file +11
-0+static struct page_info *p2m_get_clean_page (struct domain *d) +{ +struct page_info *page; + +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + page =alloc_domheap_pages (NULL ,2 ,0 );Shouldn't this allocation come from the domain's P2M pool (which is yet to be introduced)?
First, I will drop
p2m_get_clean_page()as it will be used only for p2m root page table allocation.
Quoted 7 lines
p2m_init()is called bydomain_create()[->arch_domain_create()->p2m_init()] fromcreate_domUs():
[https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L984].When
p2m_init()is called, p2m pool isn't ready and domain isn't created yet. Last one is also crucial for usage of p2m pool as p2m pool belongs to domain and thereby it is usingalloc_domheap_page(d, ...)(Not NULL as for allocation of p2m root table above), so domain should be created first.
Yet that is part of my point: This allocation should be against the domain, so it is properly accounted. What's the problem with allocating the root table when the pools is being created / filled?
Quoted 16 lines
Quoted 16 lines
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+{ +unsigned long ppn; +unsigned long hgatp_mode; +xen/arch/riscv/p2m.c line 103
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ ppn =PFN_DOWN (page_to_maddr (page_info)) &HGATP_PPN ; + +/* ASID (VMID) not supported yet */ + +#if RV_STAGE1_MODE ==SATP_MODE_SV39 + hgatp_mode =HGATP_MODE_SV39X4 ; +#elif RV_STAGE1_MODE ==SATP_MODE_SV48 + hgatp_mode =HGATP_MODE_SV48X4 ; +#else +#error "add HGATP_MODE"As before, please have the # of pre-processor directives in the first column.
Quoted 7 lines
Quoted 4 lines
xen/arch/riscv/p2m.c line 106
Quoted 3 lines
xen/arch/riscv/p2m.c new file +3
-0+#endif + +return ppn | (hgatp_mode <<HGATP_MODE_SHIFT );Use
MASK_INSR()?Do you mean
MASK_INSR(hgatp_mode, HGATP_MODE_MASK)? If yes, then I didn't get what is the point then?
The point is that generally ..._SHIFT is redundant when you also have ..._MASK; that's what MASK_EXTR() and MASK_INSR() leverage.
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/riscv/p2m.c new file +13
-0+static int p2m_alloc_table (struct domain *d) +{ +struct p2m_domain *p2m =p2m_get_hostp2m (d); + + p2m->root =p2m_allocate_root (d); +if ( !p2m->root ) +return -ENOMEM ; + + p2m->hgatp =hgatp_from_page_info (p2m->root); + +/* +* Make sure that all TLBs corresponding to the new VMID are flushed +* before using it. Quoted 8 lines
xen/arch/riscv/p2m.c line 125
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+*/ +p2m_write_lock (p2m); +p2m_force_tlb_flush_sync (p2m); +p2m_write_unlock (p2m);While Andrew directed you towards a better model in general, it won't be usable here then, as the guest didn't run on any pCPU(s) yet. Imo you want to do a single global flush e.g. when VMIDs wrap around. That'll be fewer global flushes than one per VM creation.
Quoted 1 line
I am not sure that I get a phrase 'VMIDs wrap around'.
You have to allocate them somehow. Typically you'll use the next one available. At some point you will need to start over, searching from the beginning. Prior to that now allocation of a new one will require any flush, as none of them had be in use before (after boot or the last such flush).
Quoted 6 lines
I am going to implement,
p2m_force_tlb_flush_sync()as:
static voidp2m_force_tlb_flush_sync(struct p2m_domain *p2m){ ... sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0); ...
Quoted 7 lines
}
With such implementation if the guest didn't run on any pCPU(s) yet
thend->dirty_cpumaskis empty, thensbi_remote_hfence_gvma()will do nothing
as hmask will be NULL (https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/riscv/sbi.c?ref_type=heads#L238).
I am not sure that it is a good idea as I can't find a guarantee in the spec
that TLB will be empty during boot time.
If in doubt, do one global flush while booting.
Jan
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 22 lines · Jan Beulich
Quoted 21 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich
xen/arch/riscv/p2m.c line 75
Quoted 11 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c new file +11
-0+static struct page_info *p2m_get_clean_page (struct domain *d) +{ +struct page_info *page; + +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + page =alloc_domheap_pages (NULL ,2 ,0 );Shouldn't this allocation come from the domain's P2M pool (which is yet to be introduced)?
First, I will drop
p2m_get_clean_page()as it will be used only for p2m root page table allocation.
p2m_init()is called bydomain_create()[->arch_domain_create()->p2m_init()] fromcreate_domUs():
[https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L984].
Quoted 7 lines
Quoted 4 lines
When
p2m_init()is called, p2m pool isn't ready and domain isn't created yet. Last one is also crucial for usage of p2m pool as p2m pool belongs to domain and thereby it is usingalloc_domheap_page(d, ...)(Not NULL as for allocation of p2m root table above), so domain should be created first.Yet that is part of my point: This allocation should be against the domain, so it is properly accounted. What's the problem with allocating the root table when the pools is being created / filled?
I can't use pages from pool for root table as they aren't properly aligned.
At the moment, creation of p2m pool looks like:
int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted)
{
struct page_info *pg;
ASSERT(spin_is_locked(&d->arch.paging.lock));
for ( ; ; )
{
if ( d->arch.paging.p2m_total_pages < pages )
{
/* Need to allocate more memory from domheap */
pg = alloc_domheap_page(d, MEMF_no_owner);
if ( pg == NULL )
{
printk(XENLOG_ERR "Failed to allocate P2M pages.\n");
return -ENOMEM;
}
ACCESS_ONCE(d->arch.paging.p2m_total_pages) =
d->arch.paging.p2m_total_pages + 1;
page_list_add_tail(pg, &d->arch.paging.p2m_freelist);
}
...
}
return 0; }alloc_domheap_page(d, MEMF_no_owner) allocates page table with order 0, so 4k-aligned page table. But if I needed 16k for root table and it should be 16k-aligned then I still have to use alloc_domheap_pages(NULL, 2, 0);
Or do you mean that I have to something like:
int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted)
{
struct page_info *pg;
ASSERT(spin_is_locked(&d->arch.paging.lock));+ if ( !d->arch.p2m.root )
+ {
+ unsigned int order = get_order_from_bytes(KB(16)); + unsigned int nr_pages = _AC(1,U) << order; + /* + * As mentioned in the Priviliged Architecture Spec (version 20240411) + * As explained in Section 18.5.1, for the paged virtual-memory schemes + * (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB + * and must be aligned to a 16-KiB boundary. + */ + d->arch.p2m.root = alloc_domheap_pages(d, order, MEMF_no_owner); + if ( d->arch.p2m.root == NULL )
+ panic("root page table hasn't been allocated\n");
+
+ clear_and_clean_page(d->arch.p2m.root);
+
+ /* TODO: do I need TLB flush here? */
+
+ ACCESS_ONCE(d->arch.paging.p2m_total_pages) =
+ d->arch.paging.p2m_total_pages + nr_pages;
+ }
+
...
}
(I will the current version of p2m_alloc_table() instead of open-coding.)
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+{ +unsigned long ppn; +unsigned long hgatp_mode; +xen/arch/riscv/p2m.c line 103
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ ppn =PFN_DOWN (page_to_maddr (page_info)) &HGATP_PPN ; + +/* ASID (VMID) not supported yet */ + +#if RV_STAGE1_MODE ==SATP_MODE_SV39 + hgatp_mode =HGATP_MODE_SV39X4 ; +#elif RV_STAGE1_MODE ==SATP_MODE_SV48 + hgatp_mode =HGATP_MODE_SV48X4 ; +#else +#error "add HGATP_MODE"As before, please have the # of pre-processor directives in the first column.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
xen/arch/riscv/p2m.c line 106
Quoted 3 lines
xen/arch/riscv/p2m.c new file +3
-0+#endif + +return ppn | (hgatp_mode <<HGATP_MODE_SHIFT );Use
MASK_INSR()?Do you mean
MASK_INSR(hgatp_mode, HGATP_MODE_MASK)? If yes, then I didn't get what is the point then?The point is that generally ..._SHIFT is redundant when you also have ..._MASK; that's what
MASK_EXTR()andMASK_INSR()leverage.
At the moment, there is no mask for HGATP_MODE so if to use *_MASK then I have to introduce it if it better to have *_MASK instead of *_SHIFT.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
xen/arch/riscv/p2m.c new file +7
-0+static int p2m_alloc_table (struct domain *d) +{ +struct p2m_domain *p2m =p2m_get_hostp2m (d); + + p2m->root =p2m_allocate_root (d); +if ( !p2m->root ) +return -ENOMEM ;xen/arch/riscv/p2m.c line 125
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ + p2m->hgatp =hgatp_from_page_info (p2m->root); + +/* +* Make sure that all TLBs corresponding to the new VMID are flushed +* before using it. +*/ +p2m_write_lock (p2m); +p2m_force_tlb_flush_sync (p2m); +p2m_write_unlock (p2m);While Andrew directed you towards a better model in general, it won't be usable here then, as the guest didn't run on any pCPU(s) yet. Imo you want to do a single global flush e.g. when VMIDs wrap around. That'll be fewer global flushes than one per VM creation.
Quoted 5 lines
Quoted 1 line
I am not sure that I get a phrase 'VMIDs wrap around'.
You have to allocate them somehow. Typically you'll use the next one available. At some point you will need to start over, searching from the beginning. Prior to that now allocation of a new one will require any flush, as none of them had be in use before (after boot or the last such flush).
Thanks. Now I get your point.
Won't be better to do TLB flushing during destroying of a domain so then we will be sure that TLBs connected to freed VMID aren't present in TLB anymore?
IIUC, it will work only if VMID is used, right?
In case if VMID isn't used, probably we can drop flushing here and do a flush during booting, right?
Won't be enough to flushing of guest TLB only during context switch?
Quoted 9 lines
Quoted 9 lines
I am going to implement,
p2m_force_tlb_flush_sync()as:static void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) { ... sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0); ... }
Quoted 6 lines
Quoted 5 lines
With such implementation if the guest didn't run on any pCPU(s) yet
thend->dirty_cpumaskis empty, thensbi_remote_hfence_gvma()will do nothing
as hmask will be NULL (https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/riscv/sbi.c?ref_type=heads#L238).
I am not sure that it is a good idea as I can't find a guarantee in the spec
that TLB will be empty during boot time.If in doubt, do one global flush while booting.
By booting you mean somewhere in continue_new_vcpu()?
~ Oleksii
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 94 lines · Oleksii Kurochko
Quoted 22 lines · Jan Beulich
Quoted 21 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich
xen/arch/riscv/p2m.c line 75
Quoted 11 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c new file +11
-0+static struct page_info *p2m_get_clean_page (struct domain *d) +{ +struct page_info *page; + +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + page =alloc_domheap_pages (NULL ,2 ,0 );Shouldn't this allocation come from the domain's P2M pool (which is yet to be introduced)?
First, I will drop
p2m_get_clean_page()as it will be used only for p2m root page table allocation.
p2m_init()is called bydomain_create()[->arch_domain_create()->p2m_init()] fromcreate_domUs():
[https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L984].Quoted 7 lines
Quoted 4 lines
When
p2m_init()is called, p2m pool isn't ready and domain isn't created yet. Last one is also crucial for usage of p2m pool as p2m pool belongs to domain and thereby it is usingalloc_domheap_page(d, ...)(Not NULL as for allocation of p2m root table above), so domain should be created first.Yet that is part of my point: This allocation should be against the domain, so it is properly accounted. What's the problem with allocating the root table when the pools is being created / filled?
I can't use pages from pool for root table as they aren't properly aligned.
At the moment, creation of p2m pool looks like:
intp2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted){ struct page_info *pg; ASSERT(spin_is_locked(&d->arch.paging.lock)); for ( ; ; ) { if ( d->arch.paging.p2m_total_pages < pages ) { /* Need to allocate more memory from domheap */ pg = alloc_domheap_page(d, MEMF_no_owner); if ( pg == NULL ) { printk(XENLOG_ERR "Failed to allocate P2M pages.\n"); return -ENOMEM; } ACCESS_ONCE(d->arch.paging.p2m_total_pages) = d->arch.paging.p2m_total_pages + 1; page_list_add_tail(pg, &d->arch.paging.p2m_freelist); } ... } return 0;}
alloc_domheap_page(d, MEMF_no_owner)allocates page table with order 0, so 4k-aligned page table. But if I needed 16k for root table and it should be 16k-aligned then I still have to usealloc_domheap_pages(NULL, 2, 0);Or do you mean that I have to something like:
intp2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted){ struct page_info *pg; ASSERT(spin_is_locked(&d->arch.paging.lock));xen/arch/riscv/p2m.c new file +22
-0+if ( !d->arch.p2m.root ) + { +unsigned int order =get_order_from_bytes (KB (16 )); +unsigned int nr_pages =_AC (1 ,U ) << order; +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + d->arch.p2m.root =alloc_domheap_pages (d, order, MEMF_no_owner); +if ( d->arch.p2m.root ==NULL ) +panic ("root page table hasn't been allocated\n" ); + +clear_and_clean_page (d->arch.p2m.root); + +/* TODO: do I need TLB flush here? */ + +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages + nr_pages; + } +
Quoted 1 line
}
Neither. I was thinking of you taking 4 pages off the pool in exchange for the order-2 allocation. Primarily to get the memory accounting right (or at least closer to reality).
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 4 lines
xen/arch/riscv/p2m.c new file +4
-0+{ +unsigned long ppn; +unsigned long hgatp_mode; +xen/arch/riscv/p2m.c line 103
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ ppn =PFN_DOWN (page_to_maddr (page_info)) &HGATP_PPN ; + +/* ASID (VMID) not supported yet */ + +#if RV_STAGE1_MODE ==SATP_MODE_SV39 + hgatp_mode =HGATP_MODE_SV39X4 ; +#elif RV_STAGE1_MODE ==SATP_MODE_SV48 + hgatp_mode =HGATP_MODE_SV48X4 ; +#else +#error "add HGATP_MODE"As before, please have the # of pre-processor directives in the first column.
xen/arch/riscv/p2m.c line 106
Quoted 3 lines
xen/arch/riscv/p2m.c new file +3
-0+#endif + +return ppn | (hgatp_mode <<HGATP_MODE_SHIFT );Use
MASK_INSR()?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Do you mean
MASK_INSR(hgatp_mode, HGATP_MODE_MASK)? If yes, then I didn't get what is the point then?The point is that generally ..._SHIFT is redundant when you also have ..._MASK; that's what
MASK_EXTR()andMASK_INSR()leverage.At the moment, there is no mask for
HGATP_MODEso if to use*_MASKthen I have to introduce it if it better to have*_MASKinstead of*_SHIFT.
Perhaps best to do so then.
Quoted 27 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 7 lines
xen/arch/riscv/p2m.c new file +7
-0+static int p2m_alloc_table (struct domain *d) +{ +struct p2m_domain *p2m =p2m_get_hostp2m (d); + + p2m->root =p2m_allocate_root (d); +if ( !p2m->root ) +return -ENOMEM ;xen/arch/riscv/p2m.c line 125
Quoted 10 lines
xen/arch/riscv/p2m.c new file +10
-0+ + p2m->hgatp =hgatp_from_page_info (p2m->root); + +/* +* Make sure that all TLBs corresponding to the new VMID are flushed +* before using it. +*/ +p2m_write_lock (p2m); +p2m_force_tlb_flush_sync (p2m); +p2m_write_unlock (p2m);While Andrew directed you towards a better model in general, it won't be usable here then, as the guest didn't run on any pCPU(s) yet. Imo you want to do a single global flush e.g. when VMIDs wrap around. That'll be fewer global flushes than one per VM creation.
Quoted 5 lines
Quoted 1 line
I am not sure that I get a phrase 'VMIDs wrap around'.
You have to allocate them somehow. Typically you'll use the next one available. At some point you will need to start over, searching from the beginning. Prior to that now allocation of a new one will require any flush, as none of them had be in use before (after boot or the last such flush).
Quoted 4 lines
Thanks. Now I get your point.
Won't be better to do TLB flushing during destroying of a domain so then we will be sure that TLBs connected to freed VMID aren't present in TLB anymore?
That's an option, but will result in more flushes. Furthermore there may be reasons to change the VMID for a domain while it's running.
Quoted 1 line
IIUC, it will work only if VMID is used, right?
Well, anything VMID related is of course only relevant when VMIDs are in use.
Quoted 2 lines
In case if VMID isn't used, probably we can drop flushing here and do a flush during booting, right?
That'll be too little flushing?
Quoted 1 line
Won't be enough to flushing of guest TLB only during context switch?
"only" is interesting here. Context switches are a relatively frequent operation, which in addition you want to be fast. If a flush is necessary there for correctness (e.g. when VMIDs aren't in use), you have to do it there. But if you can flush less frequently without violating correctness, you'd almost always want to use such an opportunity.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I am going to implement,
p2m_force_tlb_flush_sync()as:static void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) { ... sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0); ... }
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
With such implementation if the guest didn't run on any pCPU(s) yet
thend->dirty_cpumaskis empty, thensbi_remote_hfence_gvma()will do nothing
as hmask will be NULL (https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/riscv/sbi.c?ref_type=heads#L238).
I am not sure that it is a good idea as I can't find a guarantee in the spec
that TLB will be empty during boot time.If in doubt, do one global flush while booting.
By booting you mean somewhere in
continue_new_vcpu()?
I don't particularly mean any specific place. However, continue_new_vcpu() (by its name) isn't involved in bringing up Xen, is it?
Jan
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 90 lines · Jan Beulich
Quoted 89 lines · Oleksii Kurochko
Quoted 22 lines · Jan Beulich
Quoted 21 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich
Review
Quoted 11 lines · Oleksii Kurochko
(quoted diff) +11
-0+static struct page_info *p2m_get_clean_page(struct domain *d) +{ + struct page_info *page; + + /* + * As mentioned in the Priviliged Architecture Spec (version 20240411) + * As explained in Section 18.5.1, for the paged virtual-memory schemes + * (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB + * and must be aligned to a 16-KiB boundary. + */ + page = alloc_domheap_pages(NULL, 2, 0);Shouldn't this allocation come from the domain's P2M pool (which is yet to be introduced)?
First, I will drop
p2m_get_clean_page()as it will be used only for p2m root page table allocation.
p2m_init()is called bydomain_create()[->arch_domain_create()->p2m_init()] fromcreate_domUs():
[https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L984].Quoted 7 lines
Quoted 4 lines
When
p2m_init()is called, p2m pool isn't ready and domain isn't created yet. Last one is also crucial for usage of p2m pool as p2m pool belongs to domain and thereby it is usingalloc_domheap_page(d, ...)(Not NULL as for allocation of p2m root table above), so domain should be created first.Yet that is part of my point: This allocation should be against the domain, so it is properly accounted. What's the problem with allocating the root table when the pools is being created / filled?
I can't use pages from pool for root table as they aren't properly aligned.
At the moment, creation of p2m pool looks like:
int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) { struct page_info *pg; ASSERT(spin_is_locked(&d->arch.paging.lock)); for ( ; ; ) { if ( d->arch.paging.p2m_total_pages < pages ) { /* Need to allocate more memory from domheap */ pg = alloc_domheap_page(d, MEMF_no_owner); if ( pg == NULL ) { printk(XENLOG_ERR "Failed to allocate P2M pages.\n"); return -ENOMEM; } ACCESS_ONCE(d->arch.paging.p2m_total_pages) = d->arch.paging.p2m_total_pages + 1; page_list_add_tail(pg, &d->arch.paging.p2m_freelist); } ... } return 0; }
alloc_domheap_page(d, MEMF_no_owner)allocates page table with order 0, so 4k-aligned page table. But if I needed 16k for root table and it should be 16k-aligned then I still have to usealloc_domheap_pages(NULL, 2, 0);Or do you mean that I have to something like:
int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) { struct page_info *pg; ASSERT(spin_is_locked(&d->arch.paging.lock));(quoted diff) +19
-0+ if ( !d->arch.p2m.root ) + { + unsigned int order = get_order_from_bytes(KB(16)); + unsigned int nr_pages = _AC(1,U) << order; + /* + * As mentioned in the Priviliged Architecture Spec (version 20240411) + * As explained in Section 18.5.1, for the paged virtual-memory schemes + * (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB + * and must be aligned to a 16-KiB boundary. + */ + d->arch.p2m.root = alloc_domheap_pages(d, order, MEMF_no_owner); + if ( d->arch.p2m.root == NULL ) + panic("root page table hasn't been allocated\n"); + + clear_and_clean_page(d->arch.p2m.root); + + /* TODO: do I need TLB flush here? */ + + ACCESS_ONCE(d->arch.paging.p2m_total_pages) =
Quoted 8 lines
Quoted 5 lines
(quoted diff) +3
-0+ d->arch.paging.p2m_total_pages + nr_pages; + } +}
Neither. I was thinking of you taking 4 pages off the pool in exchange for the order-2 allocation. Primarily to get the memory accounting right (or at least closer to reality).
Do you mean that I have to call 4 times page_list_remove_head(), something like that:
xen/arch/riscv/p2m.c +38 -0
@@ -215,6 +215,44 @@ int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) } } +if ( !d->arch.p2m.root ) + { +unsigned int order =get_order_from_bytes (KB (16 )); +unsigned int nr_pages =_AC (1 ,U ) << order; + +if (ACCESS_ONCE (d->arch.paging.p2m_total_pages) < nr_pages ) +panic ("Specify more xen,domain-p2m-mem-mb\n" ); + +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + d->arch.p2m.root =alloc_domheap_pages (NULL , order,0 ); +if ( d->arch.p2m.root ==NULL ) +panic ("failed to allocate p2m root page table\n" ); + +for (unsigned int i =0 ; i < nr_pages; i++ ) + { +clear_and_clean_page (d->arch.p2m.root + i); + +/* Return memory to domheap */ + pg =page_list_remove_head (&d->arch.paging.p2m_freelist); +if ( pg ) + { +ACCESS_ONCE (d->arch.paging.p2m_total_pages)--; +free_domheap_page (pg); + } +else + { +printk (XENLOG_ERR +"Failed to free P2M pages, P2M freelist is empty.\n" ); +return -ENOMEM ; + } + } + } +return 0 ; }
Quoted 33 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
(quoted diff) +13
-0+static int p2m_alloc_table(struct domain *d) +{ + struct p2m_domain *p2m = p2m_get_hostp2m(d); + + p2m->root = p2m_allocate_root(d); + if ( !p2m->root ) + return -ENOMEM; + + p2m->hgatp = hgatp_from_page_info(p2m->root); + + /* + * Make sure that all TLBs corresponding to the new VMID are flushed + * before using it.Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ */ + p2m_write_lock(p2m); + p2m_force_tlb_flush_sync(p2m); + p2m_write_unlock(p2m);While Andrew directed you towards a better model in general, it won't be usable here then, as the guest didn't run on any pCPU(s) yet. Imo you want to do a single global flush e.g. when VMIDs wrap around. That'll be fewer global flushes than one per VM creation.
I am not sure that I get a phrase 'VMIDs wrap around'.
Quoted 8 lines
Quoted 4 lines
You have to allocate them somehow. Typically you'll use the next one available. At some point you will need to start over, searching from the beginning. Prior to that now allocation of a new one will require any flush, as none of them had be in use before (after boot or the last such flush).
Thanks. Now I get your point.
Won't be better to do TLB flushing during destroying of a domain so then we will be sure that TLBs connected to freed VMID aren't present in TLB anymore?
That's an option, but will result in more flushes. Furthermore there may be reasons to change the VMID for a domain while it's running.
Quoted 6 lines
Quoted 1 line
IIUC, it will work only if VMID is used, right?
Well, anything VMID related is of course only relevant when VMIDs are in use.
Quoted 2 lines
In case if VMID isn't used, probably we can drop flushing here and do a flush during booting, right?
That'll be too little flushing?
I meant that instead of having TLB flush in p2m_alloc_table() we could have a one flush during booting. And of course, we still should have flush on each context switch.
Quoted 7 lines
Quoted 2 lines
Won't be enough to flushing of guest TLB only during context switch?
"only" is interesting here. Context switches are a relatively frequent operation, which in addition you want to be fast. If a flush is necessary there for correctness (e.g. when VMIDs aren't in use), you have to do it there. But if you can flush less frequently without violating correctness, you'd almost always want to use such an opportunity.
Then it is better to introduce VMID now, it seems it's only one place where it should be set, when hgatp is initialized.
Does Xen have some framework to work with VMID?
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
I am going to implement,
p2m_force_tlb_flush_sync()as:static void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) { ... sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0); ... }With such implementation if the guest didn't run on any pCPU(s) yet
thend->dirty_cpumaskis empty, thensbi_remote_hfence_gvma()will do nothing
as hmask will be NULL (https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/riscv/sbi.c?ref_type=heads#L238).
I am not sure that it is a good idea as I can't find a guarantee in the spec
that TLB will be empty during boot time.
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
If in doubt, do one global flush while booting.
By booting you mean somewhere in
continue_new_vcpu()?I don't particularly mean any specific place. However,
continue_new_vcpu()(by its name) isn't involved in bringing up Xen, is it?
No, it isn't. By booting here I meant a boot of a guest domain, not Xen itself.
~ Oleksii
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 114 lines · Oleksii Kurochko
Quoted 90 lines · Jan Beulich
Quoted 89 lines · Oleksii Kurochko
Quoted 22 lines · Jan Beulich
Quoted 21 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich
xen/arch/riscv/p2m.c int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted)
Quoted 11 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +11
-0+static struct page_info *p2m_get_clean_page (struct domain *d) +{ +struct page_info *page; + +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + page =alloc_domheap_pages (NULL ,2 ,0 );Shouldn't this allocation come from the domain's P2M pool (which is yet to be introduced)?
First, I will drop
p2m_get_clean_page()as it will be used only for p2m root page table allocation.
p2m_init()is called bydomain_create()[->arch_domain_create()->p2m_init()] fromcreate_domUs():
[https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L984].Quoted 7 lines
Quoted 4 lines
When
p2m_init()is called, p2m pool isn't ready and domain isn't created yet. Last one is also crucial for usage of p2m pool as p2m pool belongs to domain and thereby it is usingalloc_domheap_page(d, ...)(Not NULL as for allocation of p2m root table above), so domain should be created first.Yet that is part of my point: This allocation should be against the domain, so it is properly accounted. What's the problem with allocating the root table when the pools is being created / filled?
I can't use pages from pool for root table as they aren't properly aligned.
At the moment, creation of p2m pool looks like:
int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) { struct page_info *pg; ASSERT(spin_is_locked(&d->arch.paging.lock)); for ( ; ; ) { if ( d->arch.paging.p2m_total_pages < pages ) { /* Need to allocate more memory from domheap */ pg = alloc_domheap_page(d, MEMF_no_owner); if ( pg == NULL ) { printk(XENLOG_ERR "Failed to allocate P2M pages.\n"); return -ENOMEM; } ACCESS_ONCE(d->arch.paging.p2m_total_pages) = d->arch.paging.p2m_total_pages + 1; page_list_add_tail(pg, &d->arch.paging.p2m_freelist); } ... } return 0; }
alloc_domheap_page(d, MEMF_no_owner)allocates page table with order 0, so 4k-aligned page table. But if I needed 16k for root table and it should be 16k-aligned then I still have to usealloc_domheap_pages(NULL, 2, 0);Or do you mean that I have to something like:
int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) { struct page_info *pg; ASSERT(spin_is_locked(&d->arch.paging.lock));xen/arch/riscv/p2m.c +19
-0+if ( !d->arch.p2m.root ) + { +unsigned int order =get_order_from_bytes (KB (16 )); +unsigned int nr_pages =_AC (1 ,U ) << order; +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + d->arch.p2m.root =alloc_domheap_pages (d, order, MEMF_no_owner); +if ( d->arch.p2m.root ==NULL ) +panic ("root page table hasn't been allocated\n" ); + +clear_and_clean_page (d->arch.p2m.root); + +/* TODO: do I need TLB flush here? */ + + ACCESS_ONCE(d->arch.paging.p2m_total_pages) =Quoted 8 lines
Quoted 5 lines
xen/arch/riscv/p2m.c +3
-0+ d->arch.paging.p2m_total_pages + nr_pages; + } +}
Neither. I was thinking of you taking 4 pages off the pool in exchange for the order-2 allocation. Primarily to get the memory accounting right (or at least closer to reality).
Do you mean that I have to call 4 times
page_list_remove_head(), something like that:xen/arch/riscv/p2m.c +5
-0@@ -215,6 +215,44 @@ int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) } } +if ( !d->arch.p2m.root ) + { +unsigned int order =get_order_from_bytes (KB (16 )); +unsigned int nr_pages =_AC (1 ,U ) << order; +
xen/arch/riscv/p2m.c int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) · line 232
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+if (ACCESS_ONCE (d->arch.paging.p2m_total_pages) < nr_pages ) +panic ("Specify more xen,domain-p2m-mem-mb\n" ); + +/* +* As mentioned in the Priviliged Architecture Spec (version 20240411) +* As explained in Section 18.5.1, for the paged virtual-memory schemes +* (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB +* and must be aligned to a 16-KiB boundary. +*/ + d->arch.p2m.root =alloc_domheap_pages (NULL , order,0 );
Imo you'd better not use NULL here, but instead pass MEMF_no_owner. See respective x86 code. I also think you want to do the freeing first, and only then do this allocation, such that ...
Quoted 9 lines
xen/arch/riscv/p2m.c +9
-0+if ( d->arch.p2m.root ==NULL ) +panic ("failed to allocate p2m root page table\n" ); + +for (unsigned int i =0 ; i < nr_pages; i++ ) + { +clear_and_clean_page (d->arch.p2m.root + i); + +/* Return memory to domheap */ + pg =page_list_remove_head (&d->arch.paging.p2m_freelist);
xen/arch/riscv/p2m.c int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) · line 251
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+if ( pg ) + { +ACCESS_ONCE (d->arch.paging.p2m_total_pages)--; +free_domheap_page (pg); + } +else + { +printk (XENLOG_ERR +"Failed to free P2M pages, P2M freelist is empty.\n" ); +return -ENOMEM ;
... this path will not have eaten more memory than was given back.
Quoted 34 lines
xen/arch/riscv/p2m.c +0
-0>>>>>> +static int p2m_alloc_table(struct domain *d) >>>>>> +{ >>>>>> + struct p2m_domain *p2m = p2m_get_hostp2m(d);Quoted 31 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
(quoted diff) +10
-0+ + p2m->root = p2m_allocate_root(d); + if ( !p2m->root ) + return -ENOMEM; + + p2m->hgatp = hgatp_from_page_info(p2m->root); + + /* + * Make sure that all TLBs corresponding to the new VMID are flushed + * before using it.Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ */ + p2m_write_lock(p2m); + p2m_force_tlb_flush_sync(p2m); + p2m_write_unlock(p2m);While Andrew directed you towards a better model in general, it won't be usable here then, as the guest didn't run on any pCPU(s) yet. Imo you want to do a single global flush e.g. when VMIDs wrap around. That'll be fewer global flushes than one per VM creation.
I am not sure that I get a phrase 'VMIDs wrap around'.
Quoted 8 lines
Quoted 4 lines
You have to allocate them somehow. Typically you'll use the next one available. At some point you will need to start over, searching from the beginning. Prior to that now allocation of a new one will require any flush, as none of them had be in use before (after boot or the last such flush).
Thanks. Now I get your point.
Won't be better to do TLB flushing during destroying of a domain so then we will be sure that TLBs connected to freed VMID aren't present in TLB anymore?
That's an option, but will result in more flushes. Furthermore there may be reasons to change the VMID for a domain while it's running.
Quoted 1 line
IIUC, it will work only if VMID is used, right?
Quoted 8 lines
Quoted 5 lines
Well, anything VMID related is of course only relevant when VMIDs are in use.
Quoted 2 lines
In case if VMID isn't used, probably we can drop flushing here and do a flush during booting, right?
That'll be too little flushing?
I meant that instead of having TLB flush in
p2m_alloc_table()we could have a one flush during booting. And of course, we still should have flush on each context switch.
Yet as said - context switches are likely too frequent for having an unconditional flush there (if it can be avoided).
Quoted 11 lines
Quoted 6 lines
Quoted 1 line
Won't be enough to flushing of guest TLB only during context switch?
"only" is interesting here. Context switches are a relatively frequent operation, which in addition you want to be fast. If a flush is necessary there for correctness (e.g. when VMIDs aren't in use), you have to do it there. But if you can flush less frequently without violating correctness, you'd almost always want to use such an opportunity.
Then it is better to introduce VMID now, it seems it's only one place where it should be set, when hgatp is initialized.
Does Xen have some framework to work with VMID?
That's all arch-specific, I think.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
I am going to implement,
p2m_force_tlb_flush_sync()as:static void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) { ... sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0); ... }With such implementation if the guest didn't run on any pCPU(s) yet
thend->dirty_cpumaskis empty, thensbi_remote_hfence_gvma()will do nothing
as hmask will be NULL (https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/riscv/sbi.c?ref_type=heads#L238).
I am not sure that it is a good idea as I can't find a guarantee in the spec
that TLB will be empty during boot time.
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
If in doubt, do one global flush while booting.
By booting you mean somewhere in
continue_new_vcpu()?I don't particularly mean any specific place. However,
continue_new_vcpu()(by its name) isn't involved in bringing up Xen, is it?No, it isn't. By booting here I meant a boot of a guest domain, not Xen itself.
Please don't call this "booting", but "starting of a guest" (or "launching" or some such). When you originally said "booting" I thought RISC-V wouldn't guarantee clean TLBs when being booted, and hence suggested to cover for this by doing a single flush during (Xen) boot. Looks like this may not be needed then, simply because of the misunderstanding.
Jan
Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 6 lines
Quoted 1 line
Won't be enough to flushing of guest TLB only during context switch?
"only" is interesting here. Context switches are a relatively frequent operation, which in addition you want to be fast. If a flush is necessary there for correctness (e.g. when VMIDs aren't in use), you have to do it there. But if you can flush less frequently without violating correctness, you'd almost always want to use such an opportunity.
Then it is better to introduce VMID now, it seems it's only one place where it should be set, when hgatp is initialized.
Does Xen have some framework to work with VMID?
That's all arch-specific, I think.
Probably, I used incorrect words to express what I want. I wrote about allocation/freeing
of VMIDs. Basically something similar to what Arm has (|p2m_vmid_allocator_init(), ||p2m_alloc_vmid(), ||p2m_free_vmid|):
https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/p2m.c?ref_type=heads#L271
It seems like it is not very arch-specific, at the moment.
But, likely, RISC-V will need to do an update|p2m_alloc_vmid(), which will do p2m's TLB flush when overflow of VMIDs happen. ~ Oleksii |
PATCH 3/6 xen/riscv: construct the P2M pages pool for guests
In reply to Oleksii KurochkoImplement p2m_set_allocation() to construct p2m pages pool for guests based on required number of pages.
This is implemented by:
- Adding a struct paging_domain which contains a freelist, a counter variable and a spinlock to struct arch_domain to indicate the free p2m pages and the number of p2m total pages in the p2m pages pool.
- Adding a helper p2m_set_allocation to set the p2m pages pool size. This helper should be called before allocating memory for a guest and is called from domain_p2m_set_allocation(), the latter is a part of common dom0less code.
--- xen/arch/riscv/include/asm/domain.h | 10 +++++ xen/arch/riscv/p2m.c | 67 +++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+)
xen/arch/riscv/include/asm/domain.h +10 -0
@@ -2,6 +2,8 @@#ifndef ASM__RISCV__DOMAIN_H #define ASM__RISCV__DOMAIN_H +#include <xen/mm.h> +#include <xen/spinlock.h> #include <xen/xmalloc.h> #include <public/hvm/params.h>
@@ -18,12 +20,20 @@ struct arch_vcpu_io {struct arch_vcpu { }; +struct paging_domain { +spinlock_t lock; +/* Free P2M pages from the pre-allocated P2M pool */ +struct page_list_head p2m_freelist; +/* Number of pages from the pre-allocated P2M pool */ +unsigned long p2m_total_pages; +};struct arch_domain {struct hvm_domain hvm;struct p2m_domain p2m; +struct paging_domain paging; };#include <xen/sched.h>
xen/arch/riscv/p2m.c +67 -0
81 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v1 3/6] xen/riscv: construct the P2M pages pool for guests
Quoted 13 lines · Oleksii Kurochko
Implement
p2m_set_allocation()to construct p2m pages pool for guests based on required number of pages.This is implemented by:
- Adding astruct paging_domainwhich contains a freelist, a counter variable and a spinlock tostruct arch_domainto indicate the free p2m pages and the number of p2m total pages in the p2m pages pool.
- Adding a helperp2m_set_allocationto set the p2m pages pool size. This helper should be called before allocating memory for a guest and is called fromdomain_p2m_set_allocation(), the latter is a part of common dom0less code.
Quoted 1 line
Signed-off-by Oleksii Kurochko <[email protected]>
As already indicated in reply to patch 2, I expect this pool will want introducing ahead of that.
xen/arch/riscv/p2m.c line 9
Quoted 12 lines
xen/arch/riscv/p2m.c +8
-0@@ -1,4 +1,12 @@#include <xen/domain_page.h> +/* +* Because of general_preempt_check() from xen/sched.h which uses +* local_events_need_delivery() but latter is declared in <asm/event.h>. +* Thereby it is needed to icnlude <xen/event.h> here before xen/sched.h. +* +* Shouldn't be xen/event.h be included in <xen/sched.h>? +*/ +#include <xen/event.h>
The question doesn't belong here; such could be put in the post-commit- message area. And the answer depends on what dependency you found missing.
Quoted 19 lines
xen/arch/riscv/p2m.c +15
-0@@ -166,3 +176,60 @@ int p2m_init(struct domain *d)return 0 ; } + +/* +* Set the pool of pages to the required number of pages. +* Returns 0 for success, non-zero for failure. +* Call with d->arch.paging.lock held. +*/ +int p2m_set_allocation (struct domain *d,unsigned long pages,bool *preempted) +{ +struct page_info *pg; + +ASSERT (spin_is_locked (&d->arch.paging.lock)); + +for ( ; ; ) + { +if ( d->arch.paging.p2m_total_pages < pages )
xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 203
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+ { +/* Need to allocate more memory from domheap */ + pg =alloc_domheap_page (d, MEMF_no_owner); +if ( pg ==NULL ) + { +printk (XENLOG_ERR "Failed to allocate P2M pages.\n" ); +return -ENOMEM ; + } +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages +1 ;
Looks like you copied this from Arm, but this code is bogus: Using ACCESS_ONCE() just on the lhs is pretty pointless. Once also used on the rhs the expression can easily become
ACCESS_ONCE(d->arch.paging.p2m_total_pages) += 1;
or even
ACCESS_ONCE(d->arch.paging.p2m_total_pages)++;
.
xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 213
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+page_list_add_tail (pg, &d->arch.paging.p2m_freelist); + } +else if ( d->arch.paging.p2m_total_pages > pages ) + { +/* Need to return memory to domheap */ + pg =page_list_remove_head (&d->arch.paging.p2m_freelist); +if ( pg ) + { +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages -1 ;
Same here then, obviously.
Quoted 4 lines
xen/arch/riscv/p2m.c +4
-0+free_domheap_page (pg); + } +else + {
xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 227
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+printk (XENLOG_ERR +"Failed to free P2M pages, P2M freelist is empty.\n" ); +return -ENOMEM ; + } + } +else +break ; + +/* Check to see if we need to yield and try again */ +if ( preempted &&general_preempt_check () )
While it's this way on both Arm and x86, I wonder how useful it is to check on every iteration, especially when freeing pages back to the buddy allocator.
Jan
Re: [PATCH v1 3/6] xen/riscv: construct the P2M pages pool for guests
Quoted 18 lines · Jan Beulich
Quoted 13 lines · Oleksii Kurochko
Implement
p2m_set_allocation()to construct p2m pages pool for guests based on required number of pages.This is implemented by:
- Adding astruct paging_domainwhich contains a freelist, a counter variable and a spinlock tostruct arch_domainto indicate the free p2m pages and the number of p2m total pages in the p2m pages pool.
- Adding a helperp2m_set_allocationto set the p2m pages pool size. This helper should be called before allocating memory for a guest and is called fromdomain_p2m_set_allocation(), the latter is a part of common dom0less code.Quoted 1 line
Signed-off-by Oleksii Kurochko<[email protected]>As already indicated in reply to patch 2, I expect this pool will want introducing ahead of that.
Quoted 14 lines
xen/arch/riscv/p2m.c line 9
Quoted 12 lines
xen/arch/riscv/p2m.c +8
-0@@ -1,4 +1,12 @@#include <xen/domain_page.h> +/* +* Because of general_preempt_check() from xen/sched.h which uses +* local_events_need_delivery() but latter is declared in <asm/event.h>. +* Thereby it is needed to icnlude <xen/event.h> here before xen/sched.h. +* +* Shouldn't be xen/event.h be included in <xen/sched.h>? +*/ +#include <xen/event.h> The question doesn't belong here; such could be put in the post-commit- message area. And the answer depends on what dependency you found missing.
It is needed for local_events_need_delivery() which is used by general_preempt_check() in p2m_set_allocation(). Otherwise, the following issue will occur:
In file included from ././include/xen/config.h:17,
from <command-line>:arch/riscv/p2m.c: In function 'p2m_set_allocation':./include/xen/sched.h:941:36: error: implicit declaration of function 'local_events_need_delivery' [-Werror=implicit-function-declaration]
941 | (!is_idle_vcpu(current) && local_events_need_delivery()) \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~./include/xen/compiler.h:26:43: note: in definition of macro 'unlikely'
26 | #define unlikely(x) __builtin_expect(!!(x),0)
| ^arch/riscv/p2m.c:244:27: note: in expansion of macro 'general_preempt_check'
244 | if ( preempted && general_preempt_check() )
| ^~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
Quoted 68 lines
Quoted 20 lines
xen/arch/riscv/p2m.c +15
-0@@ -166,3 +176,60 @@ int p2m_init(struct domain *d)return 0 ; } + +/* +* Set the pool of pages to the required number of pages. +* Returns 0 for success, non-zero for failure. +* Call with d->arch.paging.lock held. +*/ +int p2m_set_allocation (struct domain *d,unsigned long pages,bool *preempted) +{ +struct page_info *pg; + +ASSERT (spin_is_locked (&d->arch.paging.lock)); + +for ( ; ; ) + { +if ( d->arch.paging.p2m_total_pages < pages )xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 203
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+ { +/* Need to allocate more memory from domheap */ + pg =alloc_domheap_page (d, MEMF_no_owner); +if ( pg ==NULL ) + { +printk (XENLOG_ERR "Failed to allocate P2M pages.\n" ); +return -ENOMEM ; + } +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages +1 ;Looks like you copied this from Arm, but this code is bogus: Using
ACCESS_ONCE()just on the lhs is pretty pointless. Once also used on the rhs the expression can easily becomeACCESS_ONCE(d->arch.paging.p2m_total_pages) += 1;or even
ACCESS_ONCE(d->arch.paging.p2m_total_pages)++;.
xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 213
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+page_list_add_tail (pg, &d->arch.paging.p2m_freelist); + } +else if ( d->arch.paging.p2m_total_pages > pages ) + { +/* Need to return memory to domheap */ + pg =page_list_remove_head (&d->arch.paging.p2m_freelist); +if ( pg ) + { +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages -1 ;Same here then, obviously.
Quoted 4 lines
xen/arch/riscv/p2m.c +4
-0+free_domheap_page (pg); + } +else + {xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 227
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+printk (XENLOG_ERR +"Failed to free P2M pages, P2M freelist is empty.\n" ); +return -ENOMEM ; + } + } +else +break ; + +/* Check to see if we need to yield and try again */ +if ( preempted &&general_preempt_check () )
Quoted 3 lines
While it's this way on both Arm and x86, I wonder how useful it is to check on every iteration, especially when freeing pages back to the buddy allocator.
IIUC, but a preemption request could happen for both cases. And destroying of a domain could also consume long time and so not to block hypervisor if something more urgent should be handled it could be also have this check for the case of freeng pages back to the buddy allocator.
~ Oleksii
Re: [PATCH v1 3/6] xen/riscv: construct the P2M pages pool for guests
Quoted 32 lines · Oleksii Kurochko
Quoted 15 lines · Jan Beulich
xen/arch/riscv/p2m.c line 9
Quoted 12 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +8
-0@@ -1,4 +1,12 @@#include <xen/domain_page.h> +/* +* Because of general_preempt_check() from xen/sched.h which uses +* local_events_need_delivery() but latter is declared in <asm/event.h>. +* Thereby it is needed to icnlude <xen/event.h> here before xen/sched.h. +* +* Shouldn't be xen/event.h be included in <xen/sched.h>? +*/ +#include <xen/event.h> The question doesn't belong here; such could be put in the post-commit- message area. And the answer depends on what dependency you found missing.
It is needed for
local_events_need_delivery()which is used bygeneral_preempt_check()inp2m_set_allocation(). Otherwise, the following issue will occur:In file included from
././include/xen/config.h:17,
from <command-line>:arch/riscv/p2m.c: In function 'p2m_set_allocation':./include/xen/sched.h:941:36: error: implicit declaration of function 'local_events_need_delivery' [-Werror=implicit-function-declaration]941 | (!is_idle_vcpu(current) && local_events_need_delivery()) \ | ^~~~~~~~~~~~~~~~~~~~~~~~~~ ./include/xen/compiler.h:26:43: note: in definition of macro 'unlikely' 26 | #define unlikely(x) __builtin_expect(!!(x),0) | ^ arch/riscv/p2m.c:244:27: note: in expansion of macro 'general_preempt_check' 244 | if ( preempted && general_preempt_check() ) | ^~~~~~~~~~~~~~~~~~~~~
Quoted 1 line
cc1: all warnings being treated as errors
In which case my answer to your question is "No". Others may take a different perspective. (xen/sched.h being included virtually everywhere, imo we want to avoid adding dependencies there which aren't strictly necessary to keep things building.)
Quoted 67 lines
Quoted 67 lines
Quoted 19 lines
xen/arch/riscv/p2m.c +15
-0@@ -166,3 +176,60 @@ int p2m_init(struct domain *d)return 0 ; } + +/* +* Set the pool of pages to the required number of pages. +* Returns 0 for success, non-zero for failure. +* Call with d->arch.paging.lock held. +*/ +int p2m_set_allocation (struct domain *d,unsigned long pages,bool *preempted) +{ +struct page_info *pg; + +ASSERT (spin_is_locked (&d->arch.paging.lock)); + +for ( ; ; ) + { +if ( d->arch.paging.p2m_total_pages < pages )xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 203
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+ { +/* Need to allocate more memory from domheap */ + pg =alloc_domheap_page (d, MEMF_no_owner); +if ( pg ==NULL ) + { +printk (XENLOG_ERR "Failed to allocate P2M pages.\n" ); +return -ENOMEM ; + } +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages +1 ;Looks like you copied this from Arm, but this code is bogus: Using
ACCESS_ONCE()just on the lhs is pretty pointless. Once also used on the rhs the expression can easily becomeACCESS_ONCE(d->arch.paging.p2m_total_pages) += 1;or even
ACCESS_ONCE(d->arch.paging.p2m_total_pages)++;.
xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 213
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+page_list_add_tail (pg, &d->arch.paging.p2m_freelist); + } +else if ( d->arch.paging.p2m_total_pages > pages ) + { +/* Need to return memory to domheap */ + pg =page_list_remove_head (&d->arch.paging.p2m_freelist); +if ( pg ) + { +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages -1 ;Same here then, obviously.
Quoted 14 lines
xen/arch/riscv/p2m.c +14
-0+free_domheap_page (pg); + } +else + { +printk (XENLOG_ERR +"Failed to free P2M pages, P2M freelist is empty.\n" ); +return -ENOMEM ; + } + } +else +break ; + +/* Check to see if we need to yield and try again */ +if ( preempted &&general_preempt_check () )
Quoted 8 lines
Quoted 3 lines
While it's this way on both Arm and x86, I wonder how useful it is to check on every iteration, especially when freeing pages back to the buddy allocator.
IIUC, but a preemption request could happen for both cases. And destroying of a domain could also consume long time and so not to block hypervisor if something more urgent should be handled it could be also have this check for the case of freeng pages back to the buddy allocator.
The question wasn't whether to check, but how frequently. The check itself is consuming processing time, too, so one generally wants to balance the number of checks against the size of the resulting time window without any check.
Jan
Re: [PATCH v1 3/6] xen/riscv: construct the P2M pages pool for guests
Quoted 33 lines · Jan Beulich
Quoted 31 lines · Oleksii Kurochko
Quoted 15 lines · Jan Beulich
xen/arch/riscv/p2m.c line 9
Quoted 12 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +8
-0@@ -1,4 +1,12 @@#include <xen/domain_page.h> +/* +* Because of general_preempt_check() from xen/sched.h which uses +* local_events_need_delivery() but latter is declared in <asm/event.h>. +* Thereby it is needed to icnlude <xen/event.h> here before xen/sched.h. +* +* Shouldn't be xen/event.h be included in <xen/sched.h>? +*/ +#include <xen/event.h> The question doesn't belong here; such could be put in the post-commit- message area. And the answer depends on what dependency you found missing.
It is needed for
local_events_need_delivery()which is used bygeneral_preempt_check()inp2m_set_allocation(). Otherwise, the following issue will occur:In file included from
././include/xen/config.h:17,
from <command-line>:arch/riscv/p2m.c: In function 'p2m_set_allocation':./include/xen/sched.h:941:36: error: implicit declaration of function 'local_events_need_delivery' [-Werror=implicit-function-declaration]941 | (!is_idle_vcpu(current) && local_events_need_delivery()) \ | ^~~~~~~~~~~~~~~~~~~~~~~~~~ ./include/xen/compiler.h:26:43: note: in definition of macro 'unlikely' 26 | #define unlikely(x) __builtin_expect(!!(x),0) | ^ arch/riscv/p2m.c:244:27: note: in expansion of macro 'general_preempt_check' 244 | if ( preempted && general_preempt_check() ) | ^~~~~~~~~~~~~~~~~~~~~Quoted 1 line
cc1: all warnings being treated as errors
Quoted 4 lines
In which case my answer to your question is "No". Others may take a different perspective. (
xen/sched.hbeing included virtually everywhere, imo we want to avoid adding dependencies there which aren't strictly necessary to keep things building.)
Okay, then I will just update the comment and leave inclusion of xen/event.h in riscv/p2mc.
Quoted 71 lines
Quoted 71 lines
Quoted 71 lines
Quoted 20 lines
xen/arch/riscv/p2m.c +15
-0@@ -166,3 +176,60 @@ int p2m_init(struct domain *d)return 0 ; } + +/* +* Set the pool of pages to the required number of pages. +* Returns 0 for success, non-zero for failure. +* Call with d->arch.paging.lock held. +*/ +int p2m_set_allocation (struct domain *d,unsigned long pages,bool *preempted) +{ +struct page_info *pg; + +ASSERT (spin_is_locked (&d->arch.paging.lock)); + +for ( ; ; ) + { +if ( d->arch.paging.p2m_total_pages < pages )xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 203
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+ { +/* Need to allocate more memory from domheap */ + pg =alloc_domheap_page (d, MEMF_no_owner); +if ( pg ==NULL ) + { +printk (XENLOG_ERR "Failed to allocate P2M pages.\n" ); +return -ENOMEM ; + } +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages +1 ;Looks like you copied this from Arm, but this code is bogus: Using
ACCESS_ONCE()just on the lhs is pretty pointless. Once also used on the rhs the expression can easily becomeACCESS_ONCE(d->arch.paging.p2m_total_pages) += 1;or even
ACCESS_ONCE(d->arch.paging.p2m_total_pages)++;.
xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 213
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+page_list_add_tail (pg, &d->arch.paging.p2m_freelist); + } +else if ( d->arch.paging.p2m_total_pages > pages ) + { +/* Need to return memory to domheap */ + pg =page_list_remove_head (&d->arch.paging.p2m_freelist); +if ( pg ) + { +ACCESS_ONCE (d->arch.paging.p2m_total_pages) = + d->arch.paging.p2m_total_pages -1 ;Same here then, obviously.
Quoted 4 lines
xen/arch/riscv/p2m.c +4
-0+free_domheap_page (pg); + } +else + {xen/arch/riscv/p2m.c int p2m_init(struct domain *d) · line 227
Quoted 10 lines
xen/arch/riscv/p2m.c +10
-0+printk (XENLOG_ERR +"Failed to free P2M pages, P2M freelist is empty.\n" ); +return -ENOMEM ; + } + } +else +break ; + +/* Check to see if we need to yield and try again */ +if ( preempted &&general_preempt_check () )While it's this way on both Arm and x86, I wonder how useful it is to check on every iteration, especially when freeing pages back to the buddy allocator.
Quoted 7 lines
Quoted 4 lines
IIUC, but a preemption request could happen for both cases. And destroying of a domain could also consume long time and so not to block hypervisor if something more urgent should be handled it could be also have this check for the case of freeng pages back to the buddy allocator.
The question wasn't whether to check, but how frequently. The check itself is consuming processing time, too, so one generally wants to balance the number of checks against the size of the resulting time window without any check.
So you are expecting something like this:
/* Check to see if we need to yield and try again */
if ( !(d->arch.paging.p2m_total_pages % 30) && preempted &&
general_preempt_check() )
{
*preempted = true;
return -ERESTART;
}It still requires, at least, one check, but this check isn't so long as general_preempt_check() could be.
~ Oleksii
PATCH 4/6 xen/riscv: define pt_t and pt_walk_t structures
In reply to Oleksii KurochkoRefactor pte_t to be a union which hold page table entry plus pt_t and pt_walk_t structures to simpilfy p2m functions.
Also, introduce some helpers which are using pt_walk_t.
--- xen/arch/riscv/include/asm/page.h | 54 ++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-)
xen/arch/riscv/include/asm/page.h +53 -1
@@ -99,15 +99,67 @@#endif -/* Page Table entry */ typedef struct { +unsigned long v:1 ; +unsigned long r:1 ; +unsigned long w:1 ; +unsigned long x:1 ; +unsigned long u:1 ; +unsigned long g:1 ; +unsigned long a:1 ; +unsigned long d:1 ; +unsigned long rsw:2 ; +#if RV_STAGE1_MODE ==SATP_MODE_SV39 +unsigned long ppn0:9 ; +unsigned long ppn1:9 ; +unsigned long ppn2:26 ; +unsigned long rsw2:7 ; +unsigned long pbmt:2 ; +unsigned long n:1 ; +#elif RV_STAGE1_MODE ==SATP_MODE_SV48 +unsigned long ppn0:9 ; +unsigned long ppn1:9 ; +unsigned long ppn2:9 ; +unsigned long ppn3:17 ; +unsigned long rsw2:7 ; +unsigned long pbmt:2 ; +unsigned long n:1 ; +#else +#error "Add proper bits for SATP_MODE" +#endif +}pt_t ; + +typedef struct { +unsigned long rsw:10 ; +#if RV_STAGE1_MODE ==SATP_MODE_SV39 ||RV_STAGE1_MODE ==SATP_MODE_SV48 +unsigned long ppn:44 ; +#else +#error "Add proper bits for SATP_MODE" +#endif +unsigned long rsw2:10 ; +}pt_walk_t ; + +/* Page Table entry */ +typedef union {#ifdef CONFIG_RISCV_64 uint64_t pte;#else uint32_t pte;#endif +pt_t bits; +pt_walk_t walk; }pte_t ; +static inline void pte_set_mfn (pte_t *pte,mfn_t mfn) +{ + pte->walk.ppn =mfn_x (mfn); +} + +static inline mfn_t pte_get_mfn (pte_t pte) +{ +return _mfn (pte.walk.ppn); +} +static inline pte_t paddr_to_pte (paddr_t paddr,unsigned int permissions) {
Signature · 1 line
2.49.0
Re: [PATCH v1 4/6] xen/riscv: define pt_t and pt_walk_t structures
Quoted 2 lines · Oleksii Kurochko
Refactor
pte_tto be a union which hold page table entry pluspt_tandpt_walk_tstructures to simpilfy p2m functions.
Is this really simplifying things? I really view ...
Quoted 1 line
Also, introduce some helpers which are using
pt_walk_t.
... these helpers as confusing things, by using the wrong part of the union relative to what their names are. (I'll re-phrase this some at the bottom.)
With the union it's also unclear to me how to know which part of the union is the one that's valid to use, when there's no auxiliary info available.
Quoted 23 lines
xen/arch/riscv/include/asm/page.h +15
-1@@ -99,15 +99,67 @@#endif -/* Page Table entry */ typedef struct { + unsigned long v:1; + unsigned long r:1; + unsigned long w:1; + unsigned long x:1; + unsigned long u:1; + unsigned long g:1; + unsigned long a:1; + unsigned long d:1; + unsigned long rsw:2; +#if RV_STAGE1_MODE == SATP_MODE_SV39 + unsigned long ppn0:9; + unsigned long ppn1:9; + unsigned long ppn2:26; + unsigned long rsw2:7; + unsigned long pbmt:2;
xen/arch/riscv/include/asm/page.h line 127
Quoted 10 lines
xen/arch/riscv/include/asm/page.h +10
-0+ unsigned long n:1; +#elif RV_STAGE1_MODE ==SATP_MODE_SV48 + unsigned long ppn0:9; + unsigned long ppn1:9; + unsigned long ppn2:9; + unsigned long ppn3:17; + unsigned long rsw2:7; + unsigned long pbmt:2; + unsigned long n:1; +#else
Imo you want to settle on whether you want to use bitfields or #define-s to manipulate page table entries. Using a mix is going to be confusing (or worse).
xen/arch/riscv/include/asm/page.h line 135
Quoted 8 lines
xen/arch/riscv/include/asm/page.h +8
-0+#error "Add proper bits for SATP_MODE" +#endif +} pt_t; + +typedef struct { + unsigned long rsw:10; +#if RV_STAGE1_MODE == SATP_MODE_SV39 || RV_STAGE1_MODE == SATP_MODE_SV48 + unsigned long ppn: 44;
Nit: Why's there a blank after the colon here when there's none anywhere else?
xen/arch/riscv/include/asm/page.h line 161
Quoted 9 lines
xen/arch/riscv/include/asm/page.h +9
-0+static inline void pte_set_mfn (pte_t *pte,mfn_t mfn) +{ + pte->walk.ppn =mfn_x (mfn); +} + +static inline mfn_t pte_get_mfn (pte_t pte) +{ +return _mfn (pte.walk.ppn); +}
Following to my remark at the top - if you do it this way, what use are the ppn<N> fields?
Jan
Re: [PATCH v1 4/6] xen/riscv: define pt_t and pt_walk_t structures
Quoted 5 lines · Jan Beulich
Quoted 2 lines · Oleksii Kurochko
Refactor
pte_tto be a union which hold page table entry pluspt_tandpt_walk_tstructures to simpilfy p2m functions.Is this really simplifying things? I really view ...
Quoted 8 lines
Quoted 1 line
Also, introduce some helpers which are using
pt_walk_t.... these helpers as confusing things, by using the wrong part of the union relative to what their names are. (I'll re-phrase this some at the bottom.)
With the union it's also unclear to me how to know which part of the union is the one that's valid to use, when there's no auxiliary info available.
Everything is valid to use and depends on the context if it is convenient or not. It is hard to come up with a rule when and what should be used.
Quoted 29 lines
Quoted 29 lines
xen/arch/riscv/include/asm/page.h +20
-1@@ -99,15 +99,67 @@#endif -/* Page Table entry */ typedef struct { + unsigned long v:1; + unsigned long r:1; + unsigned long w:1; + unsigned long x:1; + unsigned long u:1; + unsigned long g:1; + unsigned long a:1; + unsigned long d:1; + unsigned long rsw:2; +#if RV_STAGE1_MODE == SATP_MODE_SV39 + unsigned long ppn0:9; + unsigned long ppn1:9; + unsigned long ppn2:26; + unsigned long rsw2:7; + unsigned long pbmt:2; + unsigned long n:1; +#elif RV_STAGE1_MODE == SATP_MODE_SV48 + unsigned long ppn0:9; + unsigned long ppn1:9; + unsigned long ppn2:9;
Quoted 8 lines
xen/arch/riscv/include/asm/page.h line 127
Quoted 5 lines
xen/arch/riscv/include/asm/page.h +5
-0+ unsigned long ppn3:17; + unsigned long rsw2:7; + unsigned long pbmt:2; + unsigned long n:1; +#else Imo you want to settle on whether you want to use bitfields or #define-s to manipulate page table entries. Using a mix is going to be confusing (or worse).
Generally, I am okay to use something one.
But technically it is the same things from result point of view, just different is access of a field by using a union or do a bit manipulation operations.
Quoted 11 lines
xen/arch/riscv/include/asm/page.h line 135
Quoted 9 lines
xen/arch/riscv/include/asm/page.h +8
-0+#error "Add proper bits for SATP_MODE" +#endif +} pt_t; + +typedef struct { + unsigned long rsw:10; +#if RV_STAGE1_MODE == SATP_MODE_SV39 || RV_STAGE1_MODE == SATP_MODE_SV48 + unsigned long ppn: 44;Nit: Why's there a blank after the colon here when there's none anywhere else?
Quoted 11 lines
xen/arch/riscv/include/asm/page.h line 161
Quoted 9 lines
xen/arch/riscv/include/asm/page.h +9
-0+static inline void pte_set_mfn (pte_t *pte,mfn_t mfn) +{ + pte->walk.ppn =mfn_x (mfn); +} + +static inline mfn_t pte_get_mfn (pte_t pte) +{ +return _mfn (pte.walk.ppn); +}Following to my remark at the top - if you do it this way, what use are the ppn<N> fields?
|ppn<N>| isn't actually used; it was added only to follow the PTE format specified in the architecture spec. Technically,|ppn<N>| could be merged into|ppn|, but IMO, keeping|ppn<N>| improves self-documentation of the page table format.
~ Oleksii
PATCH 5/6 xen/riscv: add new p2m types and helper macros for type classification
In reply to Oleksii Kurochko- Extended p2m_type_t with additional types: p2m_ram_ro, p2m_mmio_direct_dev, p2m_map_foreign_{rw,ro}, p2m_grant_map_{rw,ro}.
- Added macros to classify memory types: P2M_RAM_TYPES, P2M_GRANT_TYPES, P2M_FOREIGN_TYPES.
- Introduced helper predicates: p2m_is_ram(), p2m_is_foreign(), p2m_is_any_ram().
--- xen/arch/riscv/include/asm/p2m.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
xen/arch/riscv/include/asm/p2m.h +28 -0
@@ -80,8 +80,36 @@ struct p2m_domain {typedef enum { p2m_invalid =0 ,/* Nothing mapped here */ p2m_ram_rw,/* Normal read/write domain RAM */ + p2m_ram_ro,/* Read-only; writes are silently dropped */ + p2m_mmio_direct_dev,/* Read/write mapping of genuine Device MMIO area */ + p2m_map_foreign_rw,/* Read/write RAM pages from foreign domain */ + p2m_map_foreign_ro,/* Read-only RAM pages from foreign domain */ + p2m_grant_map_rw,/* Read/write grant mapping */ + p2m_grant_map_ro,/* Read-only grant mapping */ }p2m_type_t ; +/* We use bitmaps and mask to handle groups of types */ +#define p2m_to_mask (t_) BIT(t_, UL) + +/* RAM types, which map to real machine frames */ +#define P2M_RAM_TYPES (p2m_to_mask(p2m_ram_rw) | \ + p2m_to_mask(p2m_ram_ro)) + +/* Grant mapping types, which map to a real frame in another VM */ +#define P2M_GRANT_TYPES (p2m_to_mask(p2m_grant_map_rw) | \ + p2m_to_mask(p2m_grant_map_ro)) + +/* Foreign mappings types */ +#define P2M_FOREIGN_TYPES (p2m_to_mask(p2m_map_foreign_rw) | \ + p2m_to_mask(p2m_map_foreign_ro)) + +/* Useful predicates */ +#define p2m_is_ram (t_) (p2m_to_mask(t_) & P2M_RAM_TYPES) +#define p2m_is_foreign (t_) (p2m_to_mask(t_) & P2M_FOREIGN_TYPES) +#define p2m_is_any_ram (t_) (p2m_to_mask(t_) & \ + (P2M_RAM_TYPES | P2M_GRANT_TYPES | \ + P2M_FOREIGN_TYPES)) +#include <xen/p2m-common.h> static inline int get_page_and_type(struct page_info *page,
Signature · 1 line
2.49.0
Re: [PATCH v1 5/6] xen/riscv: add new p2m types and helper macros for type classification
xen/arch/riscv/include/asm/p2m.h struct p2m_domain { · line 83
Quoted 7 lines · Oleksii Kurochko
xen/arch/riscv/include/asm/p2m.h +1
-0@@ -80,8 +80,36 @@ struct p2m_domain { typedef enum { p2m_invalid = 0, /* Nothing mapped here */ p2m_ram_rw, /* Normal read/write domain RAM */ + p2m_ram_ro, /* Read-only; writes are silently dropped */
This is pretty special a type, which imo better wouldn't be introduced without there being proper support for it. (I don't suppose RISC-V hardware alone can effect this type?)
xen/arch/riscv/include/asm/p2m.h struct p2m_domain { · line 86
Quoted 3 lines
xen/arch/riscv/include/asm/p2m.h +3
-0+ p2m_mmio_direct_dev,/* Read/write mapping of genuine Device MMIO area */ + p2m_map_foreign_rw,/* Read/write RAM pages from foreign domain */ + p2m_map_foreign_ro, /* Read-only RAM pages from foreign domain */
Aiui you took these from Arm. Looking at its sole use, I'm not convinced it's used correctly. If it is, the same comment as for p2m_ram_ro above would apply here, too.
Jan
Re: [PATCH v1 5/6] xen/riscv: add new p2m types and helper macros for type classification
Quoted 11 lines · Jan Beulich
xen/arch/riscv/include/asm/p2m.h struct p2m_domain { · line 83
Quoted 7 lines · Oleksii Kurochko
xen/arch/riscv/include/asm/p2m.h +1
-0@@ -80,8 +80,36 @@ struct p2m_domain { typedef enum { p2m_invalid = 0, /* Nothing mapped here */ p2m_ram_rw, /* Normal read/write domain RAM */ + p2m_ram_ro, /* Read-only; writes are silently dropped */This is pretty special a type, which imo better wouldn't be introduced without there being proper support for it. (I don't suppose RISC-V hardware alone can effect this type?)
It is possible to make ro by using r, w, x bits of page table entry in the same way Arm does that:
case p2m_ram_ro:
e->p2m.xn = 0;
e->p2m.write = 0;
break;Quoted 7 lines
xen/arch/riscv/include/asm/p2m.h struct p2m_domain { · line 86
Quoted 4 lines
xen/arch/riscv/include/asm/p2m.h +3
-0+ p2m_mmio_direct_dev,/* Read/write mapping of genuine Device MMIO area */ + p2m_map_foreign_rw,/* Read/write RAM pages from foreign domain */ + p2m_map_foreign_ro, /* Read-only RAM pages from foreign domain */Aiui you took these from Arm. Looking at its sole use, I'm not convinced it's used correctly. If it is, the same comment as for
p2m_ram_roabove would apply here, too.
p2m_mmio_direct_dev - this one is defintely needed as it is used for device pass through to guest domain to map device's MMIO. It seems to me like it is correctly used.
Others we don't really use now in private branches but it seems like they could be useful, so I added them now.
I can drop them now and return back if such functionality which uses them will be introduced for RISC-V, and at that moment I think it will be
more clear if it is used correctly or not.
Right now, I am not sure if it is.
~ Oleksii
Re: [PATCH v1 5/6] xen/riscv: add new p2m types and helper macros for type classification
Quoted 13 lines · Oleksii Kurochko
Quoted 11 lines · Jan Beulich
xen/arch/riscv/include/asm/p2m.h struct p2m_domain { · line 83
Quoted 7 lines · Oleksii Kurochko
xen/arch/riscv/include/asm/p2m.h +1
-0@@ -80,8 +80,36 @@ struct p2m_domain { typedef enum { p2m_invalid = 0, /* Nothing mapped here */ p2m_ram_rw, /* Normal read/write domain RAM */ + p2m_ram_ro, /* Read-only; writes are silently dropped */This is pretty special a type, which imo better wouldn't be introduced without there being proper support for it. (I don't suppose RISC-V hardware alone can effect this type?)
Quoted 6 lines
It is possible to make ro by using r, w, x bits of page table entry in the same way Arm does that:
case p2m_ram_ro: e->p2m.xn = 0; e->p2m.write = 0; break;
That takes care of the r/o aspect, yes, but not of the "writes are silently dropped" one.
Quoted 10 lines
Quoted 6 lines
xen/arch/riscv/include/asm/p2m.h struct p2m_domain { · line 86
Quoted 3 lines
xen/arch/riscv/include/asm/p2m.h +3
-0+ p2m_mmio_direct_dev,/* Read/write mapping of genuine Device MMIO area */ + p2m_map_foreign_rw,/* Read/write RAM pages from foreign domain */ + p2m_map_foreign_ro, /* Read-only RAM pages from foreign domain */Aiui you took these from Arm. Looking at its sole use, I'm not convinced it's used correctly. If it is, the same comment as for
p2m_ram_roabove would apply here, too.
p2m_mmio_direct_dev- this one is defintely needed as it is used for device pass through to guest domain to map device's MMIO. It seems to me like it is correctly used.
My earlier comment was mainly about p2m_map_foreign_ro.
Quoted 6 lines
Others we don't really use now in private branches but it seems like they could be useful, so I added them now.
I can drop them now and return back if such functionality which uses them will be introduced for RISC-V, and at that moment I think it will be
more clear if it is used correctly or not.
Indeed. And maybe it'll just be p2m_map_foreign, as we have it on x86.
Jan
PATCH 6/6 xen/riscv: implement p2m mapping functionality
In reply to Oleksii KurochkoThese utilities are needed for building and managing RISC-V guest page tables and MMIO mappings by using functions map_regions_p2mt() and guest_physmap_add_entry().
To implement p2m mapping functionality the following is introduced:
- Define P2M root level/order and entry count.
- Introdude radix type for p2m types as it isn't enough free bits in pte and the helpers (p2m_type_radix_{get,set}()) to deal with them.
- Introduce p2m_is_*() helpers() as pte_is_*() helpers are checking the valid bit set in the PTE but we have to check p2m_type instead (look at the comment above p2m_is_valid() for some details).
- Introduce helper to set p2m's pte permission: p2m_set_permissions().
- Introduce helper to create p2m entry based on mfn, p2m_type_t and p2m_access_t.
- Introduce helper to generate table entry with correct attributes:
page_to_p2m_table().
- Introduce p2m page allocation function: p2m_alloc_page().
- Introduce functions to write/remove p2m's entries: p2m_{write,remove}_pte().
- Introduce function to allocate p2m table: p2m_create_table().
- Introduce functions used to free p2m entry.
- Introduce function for table walking: p2m_next_level().
- Introduce function to insert an entry in the p2m (p2m_set_entry()).
- Introduce superpage splitting: p2m_split_superpage()).
- Introduce page table type defines (PGT_{none,writable_page}, etc).
--- xen/arch/riscv/include/asm/mm.h | 32 +- xen/arch/riscv/include/asm/p2m.h | 17 +- xen/arch/riscv/include/asm/page.h | 11 + xen/arch/riscv/p2m.c | 780 ++++++++++++++++++++++++++++++ 4 files changed, 829 insertions(+), 11 deletions(-)
xen/arch/riscv/include/asm/mm.h +29 -3
@@ -12,6 +12,7 @@#include <xen/sections.h> #include <xen/types.h> +#include <asm/cmpxchg.h> #include <asm/page-bits.h> extern vaddr_t directmap_virt_start;
@@ -229,9 +230,21 @@ static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)#define PGT_writable_page PG_mask(1, 1)/* has writable mappings? */ #define PGT_type_mask PG_mask(1, 1)/* Bits 31 or 63. */ -/* Count of uses of this frame as its current type. */ -#define PGT_count_width PG_shift(2) -#define PGT_count_mask ((1UL << PGT_count_width) - 1) +/* 9-bit count of uses of this frame as its current type. */ +#define PGT_count_mask PG_mask(0x3FF, 10) + +/* +* Sv32 has 22-bit GFN. Sv{39, 48, 57} have 44-bit GFN. +* Thereby we can use for `type_info` 10 bits for all modes, having the same +* amount of bits for `type_info` for all MMU modes let us avoid introducing +* an extra #ifdef to that header: +* if we go with maximum possible bits for count on each configuration +* we would need to have a set of PGT_count_* and PGT_gfn_*). +*/ +#define PGT_gfn_width PG_shift(10) +#define PGT_gfn_mask (BIT(PGT_gfn_width, UL) - 1) + +#define PGT_INVALID_XENHEAP_GFN _gfn(PGT_gfn_mask)/* * Page needs to be scrubbed. Since this bit can only be set on a page that is
@@ -283,6 +296,19 @@ static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)#define PFN_ORDER (pg) ((pg)->v.free.order) +static inline void page_set_xenheap_gfn (struct page_info *p,gfn_t gfn) +{ +gfn_t gfn_ =gfn_eq (gfn,INVALID_GFN ) ?PGT_INVALID_XENHEAP_GFN : gfn; +unsigned long x, nx, y = p->u.inuse.type_info; + +ASSERT (is_xen_heap_page (p)); + +do { + x = y; + nx = (x & ~PGT_gfn_mask) |gfn_x (gfn_); + }while ( (y =cmpxchg (&p->u.inuse.type_info, x, nx)) != x ); +} +extern unsigned char cpu0_boot_stack[];void setup_initial_pagetables (void );
xen/arch/riscv/include/asm/p2m.h +9 -8
@@ -9,8 +9,13 @@#include <xen/rwlock.h> #include <xen/types.h> +#include <asm/page.h> #include <asm/page-bits.h> +#define P2M_ROOT_LEVEL HYP_PT_ROOT_LEVEL +#define P2M_ROOT_ORDER XEN_PT_LEVEL_ORDER(P2M_ROOT_LEVEL) +#define P2M_ROOT_PAGES (1U << P2M_ROOT_ORDER) +#define paddr_bits PADDR_BITS/* Get host p2m table */
@@ -145,14 +150,10 @@ static inline int guest_physmap_mark_populate_on_demand(struct domain *d,return -EOPNOTSUPP ; } -static inline int guest_physmap_add_entry (struct domain *d, -gfn_t gfn,mfn_t mfn, -unsigned long page_order, -p2m_type_t t) -{ -BUG_ON ("unimplemented" ); -return -EINVAL ; -} +int guest_physmap_add_entry (struct domain *d, +gfn_t gfn,mfn_t mfn, +unsigned long page_order, +p2m_type_t t);/* Untyped version for RAM only, for compatibility */ static inline int __must_check
xen/arch/riscv/include/asm/page.h +11 -0
@@ -22,6 +22,7 @@#define XEN_PT_LEVEL_SIZE (lvl) (_AT(paddr_t, 1) << XEN_PT_LEVEL_SHIFT(lvl))#define XEN_PT_LEVEL_MAP_MASK (lvl) (~(XEN_PT_LEVEL_SIZE(lvl) - 1))#define XEN_PT_LEVEL_MASK (lvl) (VPN_MASK << XEN_PT_LEVEL_SHIFT(lvl)) +#define XEN_PT_ENTRIES (_AT(unsigned int, 1) << PAGETABLE_ORDER)/* * PTE format:
@@ -69,10 +70,20 @@#define PTE_PMBT_NOCACHE BIT(61, UL)#define PTE_PMBT_IO BIT(62, UL) +enum pmbt_type_t { + pbmt_pma, + pbmt_nc, + pbmt_io, + pbmt_rsvd, + pbmt_max, +}; +#define PTE_ACCESS_MASK (PTE_READABLE | PTE_WRITABLE | PTE_EXECUTABLE)#define PTE_PBMT_MASK (PTE_PMBT_NOCACHE | PTE_PMBT_IO) +#define P2M_CLEAR_PERM (p2m_pte) ((p2m_pte).pte & ~PTE_ACCESS_MASK) +/* Calculate the offsets into the pagetables for a given VA */ #define pt_linear_offset (lvl, va) ((va) >> XEN_PT_LEVEL_SHIFT(lvl))
xen/arch/riscv/p2m.c +780 -0
789 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v1 6/6] xen/riscv: implement p2m mapping functionality
Quoted 11 lines · Oleksii Kurochko
These utilities are needed for building and managing RISC-V guest page tables and MMIO mappings by using functions
map_regions_p2mt()andguest_physmap_add_entry().To implement p2m mapping functionality the following is introduced:
- Define P2M root level/order and entry count.
- Introdude radix type for p2m types as it isn't enough free bits in pte and the helpers (p2m_type_radix_{get,set}()) to deal with them.
- Introduce p2m_is_*()helpers()as pte_is_*() helpers are checking the valid bit set in the PTE but we have to checkp2m_typeinstead (look at the comment abovep2m_is_valid()for some details).
May I suggest to name them at least p2me_is_*() then, as they check entries rather than entire P2Ms? Same perhaps elsewhere.
Quoted 14 lines
- Introduce helper to set p2m's pte permission:
p2m_set_permissions().
- Introduce helper to create p2m entry based on mfn,p2m_type_tandp2m_access_t.
- Introduce helper to generate table entry with correct attributes:
page_to_p2m_table().
- Introduce p2m page allocation function:p2m_alloc_page().
- Introduce functions to write/remove p2m's entries: p2m_{write,remove}_pte().
- Introduce function to allocate p2m table:p2m_create_table().
- Introduce functions used to free p2m entry.
- Introduce function for table walking:p2m_next_level().
- Introduce function to insert an entry in the p2m (p2m_set_entry()).
- Introduce superpage splitting:p2m_split_superpage()).
- Introduce page table type defines (PGT_{none,writable_page}, etc).
Quoted 7 lines
Signed-off-by Oleksii Kurochko <[email protected]>--- xen/arch/riscv/include/asm/mm.h | 32 +- xen/arch/riscv/include/asm/p2m.h | 17 +- xen/arch/riscv/include/asm/page.h | 11 + xen/arch/riscv/p2m.c | 780 ++++++++++++++++++++++++++++++ 4 files changed, 829 insertions(+), 11 deletions(-)
It's imo too many things you do in one go here.
Jan
Re: [PATCH v1 6/6] xen/riscv: implement p2m mapping functionality
Quoted 12 lines · Jan Beulich
Quoted 11 lines · Oleksii Kurochko
These utilities are needed for building and managing RISC-V guest page tables and MMIO mappings by using functions
map_regions_p2mt()andguest_physmap_add_entry().To implement p2m mapping functionality the following is introduced:
- Define P2M root level/order and entry count.
- Introdude radix type for p2m types as it isn't enough free bits in pte and the helpers (p2m_type_radix_{get,set}()) to deal with them.
- Introduce p2m_is_*()helpers()as pte_is_*() helpers are checking the valid bit set in the PTE but we have to checkp2m_typeinstead (look at the comment abovep2m_is_valid()for some details).
Quoted 2 lines
May I suggest to name them at least p2me_is_*() then, as they check entries rather than entire P2Ms? Same perhaps elsewhere.
Sure, I will handle that during a work on v2.
Quoted 15 lines
Quoted 15 lines
- Introduce helper to set p2m's pte permission:
p2m_set_permissions().
- Introduce helper to create p2m entry based on mfn,p2m_type_tandp2m_access_t.
- Introduce helper to generate table entry with correct attributes:page_to_p2m_table().- Introduce p2m page allocation function:
p2m_alloc_page().
- Introduce functions to write/remove p2m's entries: p2m_{write,remove}_pte().
- Introduce function to allocate p2m table:p2m_create_table().
- Introduce functions used to free p2m entry.
- Introduce function for table walking:p2m_next_level().
- Introduce function to insert an entry in the p2m (p2m_set_entry()).
- Introduce superpage splitting:p2m_split_superpage()).
- Introduce page table type defines (PGT_{none,writable_page}, etc).
Quoted 8 lines
Quoted 7 lines
Signed-off-by Oleksii Kurochko<[email protected]>--- xen/arch/riscv/include/asm/mm.h | 32 +- xen/arch/riscv/include/asm/p2m.h | 17 +- xen/arch/riscv/include/asm/page.h | 11 + xen/arch/riscv/p2m.c | 780 ++++++++++++++++++++++++++++++ 4 files changed, 829 insertions(+), 11 deletions(-)It's imo too many things you do in one go here.
I will split to smaller patches.
Thanks.
~ Oleksii