xen/riscv: introduce identity mapping
cover xen/riscv: introduce identity mapping
The patch series introduces things necessary to implement identity mapping:
1. Make identity mapping for _start and stack.
2. Enable MMU.
3. Jump to the virtual address world
4. Remove identity mapping for _start and stack.
Also current patch series introduces the calculation of physical offset before MMU is enabled as access to physical offset will be calculated wrong after MMU will be enabled because access to phys_off variable is PC-relative and in the case when linker address != load address, it will cause MMU fault.
One more thing that was done is:
* Added SPDX tags.
* Added __ASSEMBLY__ guards.
* move extern of cpu0_boot_stack to a header.
The reason for this patch series can be found here:
https://lore.kernel.org/xen-devel/[email protected]/
Oleksii Kurochko (8):
xen/riscv: make sure that identity mapping isn't bigger then page size xen/riscv: add .sbss section to .bss
xen/riscv: introduce reset_stack() function
xen/riscv: introduce function for physical offset calculation xen/riscv: introduce identity mapping
xen/riscv: add SPDX tags
xen/riscv: add __ASSEMBLY__ guards
xen/riscv: move extern of cpu0_boot_stack to header
xen/arch/riscv/include/asm/config.h | 2 + xen/arch/riscv/include/asm/current.h | 2 + xen/arch/riscv/include/asm/early_printk.h | 2 + xen/arch/riscv/include/asm/mm.h | 9 +- xen/arch/riscv/include/asm/page-bits.h | 2 + xen/arch/riscv/include/asm/page.h | 6 ++ xen/arch/riscv/include/asm/traps.h | 2 + xen/arch/riscv/include/asm/types.h | 2 + xen/arch/riscv/mm.c | 119 +++++++++++++++------- xen/arch/riscv/riscv64/head.S | 40 +++++++- xen/arch/riscv/setup.c | 16 +-- xen/arch/riscv/xen.lds.S | 11 +- 12 files changed, 160 insertions(+), 53 deletions(-)
Signature · 1 line
2.40.1
PATCH 1/8 xen/riscv: make sure that identity mapping isn't bigger then page size
--- xen/arch/riscv/xen.lds.S | 9 +++++++++ 1 file changed, 9 insertions(+)
xen/arch/riscv/xen.lds.S +9 -0
@@ -20,6 +20,7 @@ SECTIONS . = XEN_VIRT_START; _start = .; .text : { + _idmap_start = .; _stext = .; /* Text section */ *(.text.header)
@@ -35,6 +36,7 @@ SECTIONS *(.gnu.warning) . = ALIGN(POINTER_ALIGN); _etext = .; /* End of text section */ + _idmap_end = .; } :text . = ALIGN(PAGE_SIZE);
@@ -174,3 +176,10 @@ ASSERT(!SIZEOF(.got), ".got non-empty") ASSERT(!SIZEOF(.got.plt), ".got.plt non-empty") ASSERT(_end - _start <= MB(2), "Xen too large for early-boot assumptions") + +/* + * We require that Xen is loaded at a page boundary, so this ensures that any + * code running on the identity map cannot cross a page boundary. + */ +ASSERT(IS_ALIGNED(_idmap_start, PAGE_SIZE), "_idmap_start should be page-aligned") +ASSERT(_idmap_end - _idmap_start <= PAGE_SIZE, "Identity mapped code is larger than a page size")
Signature · 1 line
2.40.1
Re: [PATCH v1 1/8] xen/riscv: make sure that identity mapping isn't bigger then page size
Quoted 2 lines · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>
Alistair
Quoted 39 lines
--- xen/arch/riscv/xen.lds.S | 9 +++++++++ 1 file changed, 9 insertions(+)xen/arch/riscv/xen.lds.S +9
-1@@ -20,6 +20,7 @@ SECTIONS . = XEN_VIRT_START; _start = .; .text : { + _idmap_start = .; _stext = .; /* Text section */ *(.text.header)@@ -35,6 +36,7 @@ SECTIONS *(.gnu.warning) . = ALIGN(POINTER_ALIGN); _etext = .; /* End of text section */ + _idmap_end = .; } :text . = ALIGN(PAGE_SIZE);@@ -174,3 +176,10 @@ ASSERT(!SIZEOF(.got), ".got non-empty") ASSERT(!SIZEOF(.got.plt), ".got.plt non-empty") ASSERT(_end - _start <= MB(2), "Xen too large for early-boot assumptions") + +/* + * We require that Xen is loaded at a page boundary, so this ensures that any + * code running on the identity map cannot cross a page boundary. + */ +ASSERT(IS_ALIGNED(_idmap_start, PAGE_SIZE), "_idmap_start should be page-aligned") +ASSERT(_idmap_end - _idmap_start <= PAGE_SIZE, "Identity mapped code is larger than a page size") --2.40.1
Re: [PATCH v1 1/8] xen/riscv: make sure that identity mapping isn't bigger then page size
In reply to Oleksii KurochkoQuoted 1 line · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>
Such commits without description are worrying. This may be okay for entirely trivial and obvious changes, but that's going to be the exception.
Quoted 11 lines
xen/arch/riscv/xen.lds.S +1
-0@@ -20,6 +20,7 @@ SECTIONS . = XEN_VIRT_START; _start = .; .text : { + _idmap_start = .; _stext = .; /* Text section */ *(.text.header)
xen/arch/riscv/xen.lds.S SECTIONS · line 40
Quoted 5 lines
xen/arch/riscv/xen.lds.S +1
-0@@ -35,6 +36,7 @@ SECTIONS *(.gnu.warning) . = ALIGN(POINTER_ALIGN); _etext = .; /* End of text section */ + _idmap_end = .; } :text
So this covers all of .text. Why is it expected that .text will be (and remain) ...
xen/arch/riscv/xen.lds.S ASSERT(!SIZEOF(.got), ".got non-empty") · line 185
Quoted 11 lines
xen/arch/riscv/xen.lds.S +7
-0@@ -174,3 +176,10 @@ ASSERT(!SIZEOF(.got), ".got non-empty") ASSERT(!SIZEOF(.got.plt), ".got.plt non-empty") ASSERT(_end - _start <= MB(2), "Xen too large for early-boot assumptions") + +/* + * We require that Xen is loaded at a page boundary, so this ensures that any + * code running on the identity map cannot cross a page boundary. + */ +ASSERT(IS_ALIGNED(_idmap_start, PAGE_SIZE), "_idmap_start should be page-aligned") +ASSERT(_idmap_end - _idmap_start <= PAGE_SIZE, "Identity mapped code is larger than a page size")
... less than 4k in size? And why is only .text of interest, but not other sections?
I find the other assertion a little puzzling too: Isn't that merely checking that XEN_VIRT_START is page aligned?
Jan
Re: [PATCH v1 1/8] xen/riscv: make sure that identity mapping isn't bigger then page size
Quoted 38 lines · Jan Beulich
Quoted 1 line · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>Such commits without description are worrying. This may be okay for entirely trivial and obvious changes, but that's going to be the exception.
Quoted 13 lines
xen/arch/riscv/xen.lds.S +1
-0@@ -20,6 +20,7 @@ SECTIONS . = XEN_VIRT_START; _start = .; .text : { + _idmap_start = .;
_stext= .; /* Text section */ *(.text.header)xen/arch/riscv/xen.lds.S +0
-0@@ -35,6 +36,7 @@ SECTIONS *(.gnu.warning) . = ALIGN(POINTER_ALIGN);Quoted 3 lines
_etext= .; /* End of text section */ +_idmap_end= .;
} :textSo this covers all of .text. Why is it expected that .text will be (and
remain) ...Quoted 10 lines
xen/arch/riscv/xen.lds.S +0
-0@@ -174,3 +176,10 @@ ASSERT(!SIZEOF(.got), ".got non-empty") ASSERT(!SIZEOF(.got.plt), ".got.plt non-empty")
ASSERT(_end -_start<= MB(2), "Xen too large for early-boot assumptions")xen/arch/riscv/xen.lds.S +3
-0+ +/* + * We require that Xen is loaded at a page boundary, so thisensures that any
+ * code running on the identity map cannot cross a page boundary.
Quoted 8 lines
Quoted 5 lines
+ */
+ASSERT(IS_ALIGNED(_idmap_start,PAGE_SIZE), "_idmap_startshould be page-aligned")
+ASSERT(_idmap_end -_idmap_start<=PAGE_SIZE, "Identity mapped code is larger than a page size")... less than 4k in size? And why is only .text of interest, but not other sections?
An idea was to keep identity mapping as small as possible because basically identity mapping is needed only for a few instructions. (probably it will be better to create a separate section and put all necessary functions there)
Another point was to map the necessary code for switching from 1:1 mapping in one cycle. ( we are using 4K as a page size )
But it looks like PAGE_SIZE isn't enough. I rebased all my patches that are needed to run Dom0 and compiler complains that _idmap is bigger than PAGE_SIZE so I probably have to reject this idea ( to map only PAGE_SIZE ).
Actually not only .text section is needed but also stack should be 1:1 mapped. ( what is done in setup_initial_pagetables() )
Quoted 3 lines
I find the other assertion a little puzzling too: Isn't that merely checking that
XEN_VIRT_STARTis page aligned?
Yeah, you are right.
~ Oleksii
PATCH 2/8 xen/riscv: add .sbss section to .bss
In reply to Oleksii KurochkoSometimes variables are located in .sbss section but it won't be mapped after MMU will be enabled.
To avoid MMU failures .sbss should be mapped
--- xen/arch/riscv/xen.lds.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/riscv/xen.lds.S +1 -1
@@ -151,7 +151,7 @@ SECTIONS *(.bss.percpu.read_mostly) . = ALIGN(SMP_CACHE_BYTES); __per_cpu_data_end = .; - *(.bss .bss.*) + *(.bss .bss.* .sbss) . = ALIGN(POINTER_ALIGN); __bss_end = .; } :text
Signature · 1 line
2.40.1
Re: [PATCH v1 2/8] xen/riscv: add .sbss section to .bss
Quoted 6 lines · Oleksii Kurochko
Sometimes variables are located in .sbss section but it won't be mapped after MMU will be enabled.
To avoid MMU failures .sbss should be mappedSigned-off-by Oleksii Kurochko <[email protected]>
Alistair
Quoted 21 lines
--- xen/arch/riscv/xen.lds.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)xen/arch/riscv/xen.lds.S +1
-2@@ -151,7 +151,7 @@ SECTIONS *(.bss.percpu.read_mostly) . = ALIGN(SMP_CACHE_BYTES); __per_cpu_data_end = .; - *(.bss .bss.*) + *(.bss .bss.* .sbss) . = ALIGN(POINTER_ALIGN); __bss_end = .; } :text --2.40.1
Re: [PATCH v1 2/8] xen/riscv: add .sbss section to .bss
In reply to Oleksii KurochkoQuoted 14 lines · Oleksii Kurochko
Sometimes variables are located in .sbss section but it won't be mapped after MMU will be enabled.
To avoid MMU failures .sbss should be mappedSigned-off-by Oleksii Kurochko <[email protected]>--- xen/arch/riscv/xen.lds.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)xen/arch/riscv/xen.lds.S +0
-0
xen/arch/riscv/xen.lds.S SECTIONS · line 157
Quoted 8 lines
xen/arch/riscv/xen.lds.S +1
-1@@ -151,7 +151,7 @@ SECTIONS *(.bss.percpu.read_mostly) . = ALIGN(SMP_CACHE_BYTES); __per_cpu_data_end = .; - *(.bss .bss.*) + *(.bss .bss.* .sbss) . = ALIGN(POINTER_ALIGN); __bss_end = .; } :text
Two remarks, despite Alistair's ack: Wouldn't it be better to add .sbss.* right away as well? And strictly speaking wouldn't it be more logical to have .sbss ahead of .bss?
Jan
Re: [PATCH v1 2/8] xen/riscv: add .sbss section to .bss
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Oleksii Kurochko
Sometimes variables are located in .sbss section but it won't be mapped after MMU will be enabled.
To avoid MMU failures .sbss should be mappedSigned-off-by Oleksii Kurochko <[email protected]>---
xen/arch/riscv/xen.lds.S | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)xen/arch/riscv/xen.lds.S +1
-1@@ -151,7 +151,7 @@ SECTIONS *(.bss.percpu.read_mostly) . = ALIGN(SMP_CACHE_BYTES); __per_cpu_data_end = .; - *(.bss .bss.*) + *(.bss .bss.* .sbss) . = ALIGN(POINTER_ALIGN);
Quoted 8 lines
xen/arch/riscv/xen.lds.S SECTIONS · line 157
Quoted 2 lines
xen/arch/riscv/xen.lds.S +0
-0__bss_end = .; } :textTwo remarks, despite Alistair's ack: Wouldn't it be better to add .sbss.*
right away as well? And strictly speaking wouldn't it be more logical to
have .sbss ahead of .bss?
It makes sense. I'll take it into account in the following patch series.
Thanks.
Quoted 2 lines
Jan
PATCH 3/8 xen/riscv: introduce reset_stack() function
In reply to Oleksii Kurochko--- xen/arch/riscv/riscv64/head.S | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
xen/arch/riscv/riscv64/head.S +7 -1
@@ -27,8 +27,14 @@ ENTRY(start) add t3, t3, __SIZEOF_POINTER__ bltu t3, t4, .L_clear_bss + jal reset_stack + + tail start_xen + +ENTRY(reset_stack) la sp, cpu0_boot_stack li t0, STACK_SIZE add sp, sp, t0 - tail start_xen + ret +
Signature · 1 line
2.40.1
Re: [PATCH v1 3/8] xen/riscv: introduce reset_stack() function
Quoted 2 lines · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>
Alistair
Quoted 28 lines
--- xen/arch/riscv/riscv64/head.S | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)xen/arch/riscv/riscv64/head.S +7
-2@@ -27,8 +27,14 @@ ENTRY(start) add t3, t3, __SIZEOF_POINTER__ bltu t3, t4, .L_clear_bss + jal reset_stack + + tail start_xen + +ENTRY(reset_stack) la sp, cpu0_boot_stack li t0, STACK_SIZE add sp, sp, t0 - tail start_xen + ret + --2.40.1
Re: [PATCH v1 3/8] xen/riscv: introduce reset_stack() function
In reply to Oleksii KurochkoQuoted 1 line · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>
This wants addressing the "Why?" aspect in the description. Is the present code wrong in some perhaps subtle way? Are you meaning to re-use the code? If so, in which way (which is relevant to determine whether the new function may actually continue to live in .text.header)?
Jan
Quoted 18 lines
xen/arch/riscv/riscv64/head.S +7
-1@@ -27,8 +27,14 @@ ENTRY(start) add t3, t3, __SIZEOF_POINTER__ bltu t3, t4, .L_clear_bss + jal reset_stack + + tail start_xen + +ENTRY(reset_stack) la sp, cpu0_boot_stack li t0, STACK_SIZE add sp, sp, t0 - tail start_xen + ret +
Re: [PATCH v1 3/8] xen/riscv: introduce reset_stack() function
Quoted 7 lines · Jan Beulich
Quoted 1 line · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>This wants addressing the "Why?" aspect in the description. Is the present
code wrong in some perhaps subtle way? Are you meaning to re-use the code?
I am re-using it after switching from 1:1 world to reset a stack.
Quoted 2 lines
If so, in which way (which is relevant to determine whether the new function may actually continue to live in .text.header)?
Actually there is no such requirement for reset_stack to live in .text.header.
Quoted 10 lines
Quoted 10 lines
xen/arch/riscv/riscv64/head.S +0
-0@@ -27,8 +27,14 @@ ENTRY(start)add t3, t3,
__SIZEOF_POINTER__
bltu t3, t4, .L_clear_bss
+ jalreset_stackxen/arch/riscv/riscv64/head.S +3
-0+ + tail start_xen +
Quoted 9 lines
Quoted 8 lines
+ENTRY(reset_stack)
la sp,cpu0_boot_stack
li t0,STACK_SIZE
add sp, sp, t0
- tailstart_xen
+ ret
+
~ Oleksii
Re: [PATCH v1 3/8] xen/riscv: introduce reset_stack() function
In reply to Jan BeulichQuoted 10 lines · Jan Beulich
Quoted 1 line · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>This wants addressing the "Why?" aspect in the description. Is the present
code wrong in some perhaps subtle way? Are you meaning to re-use the code?
If so, in which way (which is relevant to determine whether the new function may actually continue to live in .text.header)?
As I mentioned in previous e-mail there is no such requirement for reset_stack() function to live in .text.header.
I think such requirement exists only for _start() function as we would like to see it at the start of image as a bootloader jumps to it and it is expected to be the first instructions.
Even though I don't see any issue for reset_stack() to live in .text.header section, should it be moved to .text section? If yes, I would appreciate hearing why it would be better.
~ Oleksii
Re: [PATCH v1 3/8] xen/riscv: introduce reset_stack() function
Quoted 14 lines · Oleksii
Quoted 10 lines · Jan Beulich
Quoted 1 line · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>This wants addressing the "Why?" aspect in the description. Is the present
code wrong in some perhaps subtle way? Are you meaning to re-use the code?
If so, in which way (which is relevant to determine whether the new function may actually continue to live in .text.header)?As I mentioned in previous e-mail there is no such requirement for
reset_stack()function to live in .text.header.
Quoted 7 lines
I think such requirement exists only for
_start()function as we would like to see it at the start of image as a bootloader jumps to it and it is expected to be the first instructions.Even though I don't see any issue for
reset_stack()to live in .text.header section, should it be moved to .text section? If yes, I would appreciate hearing why it would be better.
Because of the simple principle of special sections better only holding what really needs to be there.
Jan
PATCH 4/8 xen/riscv: introduce function for physical offset calculation
In reply to Oleksii KurochkoThe function was introduced to not calculate and save physical offset before MMU is enabled because access to start() is PC-relative and in case of linker_addr != load_addr it will result in incorrect value in phys_offset.
--- xen/arch/riscv/include/asm/mm.h | 2 ++ xen/arch/riscv/mm.c | 18 +++++++++++++++--- xen/arch/riscv/riscv64/head.S | 2 ++ 3 files changed, 19 insertions(+), 3 deletions(-)
xen/arch/riscv/include/asm/mm.h +2 -0
@@ -11,4 +11,6 @@ void setup_initial_pagetables(void);void enable_mmu (void );void cont_after_mmu_is_enabled (void ); +void calc_phys_offset (void ); +#endif /* _ASM_RISCV_MM_H */
xen/arch/riscv/mm.c +15 -3
@@ -19,9 +19,11 @@ struct mmu_desc {extern unsigned char cpu0_boot_stack[STACK_SIZE ]; -#define PHYS_OFFSET ((unsigned long)_start - XEN_VIRT_START) -#define LOAD_TO_LINK (addr) ((addr) - PHYS_OFFSET) -#define LINK_TO_LOAD (addr) ((addr) + PHYS_OFFSET) +static unsigned long phys_offset; + +#define LOAD_TO_LINK (addr) ((unsigned long)(addr) - phys_offset) +#define LINK_TO_LOAD (addr) ((unsigned long)(addr) + phys_offset) +/* * It is expected that Xen won't be more then 2 MB.
@@ -273,3 +275,13 @@ void __init noreturn noinline enable_mmu()switch_stack_and_jump ((unsigned long )cpu0_boot_stack +STACK_SIZE , cont_after_mmu_is_enabled); } + +/* +* calc_phys_offset() should be used before MMU is enabled because access to +* start() is PC-relative and in case when load_addr != linker_addr phys_offset +* will have an incorrect value +*/ +void calc_phys_offset (void ) +{ + phys_offset = (unsigned long )start -XEN_VIRT_START ; +}
xen/arch/riscv/riscv64/head.S +2 -0
@@ -29,6 +29,8 @@ ENTRY(start) jal reset_stack + jal calc_phys_offset + tail start_xen ENTRY(reset_stack)
Signature · 1 line
2.40.1
Re: [PATCH v1 4/8] xen/riscv: introduce function for physical offset calculation
Quoted 4 lines · Oleksii Kurochko
The function was introduced to not calculate and save physical offset before MMU is enabled because access to
start()is PC-relative and in case oflinker_addr!=load_addrit will result in incorrect value inphys_offset.
"... to _not_ calculate ..."?
xen/arch/riscv/mm.c struct mmu_desc { · line 22
Quoted 10 lines
xen/arch/riscv/mm.c +1
-3@@ -19,9 +19,11 @@ struct mmu_desc {extern unsigned char cpu0_boot_stack[STACK_SIZE ]; -#define PHYS_OFFSET ((unsigned long)_start - XEN_VIRT_START) -#define LOAD_TO_LINK (addr) ((addr) - PHYS_OFFSET) -#define LINK_TO_LOAD (addr) ((addr) + PHYS_OFFSET) +static unsigned long phys_offset;
__ro_after_init?
xen/arch/riscv/mm.c struct mmu_desc { · line 27
Quoted 4 lines
xen/arch/riscv/mm.c +3
-0+#define LOAD_TO_LINK (addr) ((unsigned long)(addr) - phys_offset) +#define LINK_TO_LOAD (addr) ((unsigned long)(addr) + phys_offset) +
Nit: No double blank lines please.
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 284
Quoted 11 lines
xen/arch/riscv/mm.c +7
-0@@ -273,3 +275,13 @@ void __init noreturn noinline enable_mmu()switch_stack_and_jump ((unsigned long )cpu0_boot_stack +STACK_SIZE , cont_after_mmu_is_enabled); } + +/* +* calc_phys_offset() should be used before MMU is enabled because access to +* start() is PC-relative and in case when load_addr != linker_addr phys_offset +* will have an incorrect value +*/ +void calc_phys_offset (void )
__init? And nit: No double blanks please.
Jan
Re: [PATCH v1 4/8] xen/riscv: introduce function for physical offset calculation
Quoted 7 lines · Jan Beulich
Quoted 4 lines · Oleksii Kurochko
The function was introduced to not calculate and save physical offset before MMU is enabled because access to
start()is PC-relative and in case oflinker_addr!=load_addrit will result in incorrect value inphys_offset."... to _not_ calculate ..."?
_not_ should be dropped. Thanks.
Quoted 13 lines
xen/arch/riscv/mm.c struct mmu_desc { · line 22
Quoted 11 lines
xen/arch/riscv/mm.c +0
-0@@ -19,9 +19,11 @@ struct mmu_desc {
extern unsigned char cpu0_boot_stack[STACK_SIZE];xen/arch/riscv/mm.c +1
-3-#define PHYS_OFFSET ((unsigned long)_start - XEN_VIRT_START) -#define LOAD_TO_LINK (addr) ((addr) - PHYS_OFFSET) -#define LINK_TO_LOAD (addr) ((addr) + PHYS_OFFSET) +static unsigned long phys_offset;
__ro_after_init?
It makes sense to mark variable as __ro_after_init. I'll take into account in new version of patch.
Quoted 7 lines
Quoted 5 lines
xen/arch/riscv/mm.c +3
-0+#define LOAD_TO_LINK (addr) ((unsigned long)(addr) - phys_offset) +#define LINK_TO_LOAD (addr) ((unsigned long)(addr) + phys_offset) +Nit: No double blank lines please.
Sorry. Missed that.
Quoted 17 lines
Quoted 9 lines
xen/arch/riscv/mm.c +0
-0@@ -273,3 +275,13 @@ void __init noreturn noinline enable_mmu()switch_stack_and_jump((unsigned long)cpu0_boot_stack +
STACK_SIZE,
cont_after_mmu_is_enabled); }xen/arch/riscv/mm.c +3
-0+ +/* +* calc_phys_offset() should be used before MMU is enabled because Quoted 6 lines
access to
+ *start()is PC-relative and in case whenload_addr!=linker_addrphys_offset
+ * will have an incorrect value
+ */
+voidcalc_phys_offset(void)
__init? And nit: No double blanks please.
Thanks. It should be __init. I'll remove double blanks in new patch version.
~ Oleksii
PATCH 5/8 xen/riscv: introduce identity mapping
In reply to Oleksii KurochkoThe way how switch to virtual address was implemented in the commit e66003e7be ("xen/riscv: introduce setup_initial_pages") wasn't safe enough so identity mapping was introduced and used.
--- xen/arch/riscv/include/asm/mm.h | 3 +- xen/arch/riscv/mm.c | 99 ++++++++++++++++++++++----------- xen/arch/riscv/riscv64/head.S | 30 ++++++++++ xen/arch/riscv/setup.c | 14 +---- 4 files changed, 99 insertions(+), 47 deletions(-)
xen/arch/riscv/include/asm/mm.h +2 -1
@@ -9,7 +9,8 @@void setup_initial_pagetables (void );void enable_mmu (void ); -void cont_after_mmu_is_enabled (void ); + +void remove_identity_mapping (void );void calc_phys_offset (void );
xen/arch/riscv/mm.c +65 -34
148 lines not on this page · raw message
xen/arch/riscv/riscv64/head.S +30 -0
@@ -31,6 +31,36 @@ ENTRY(start) jal calc_phys_offset + jal setup_initial_pagetables + + jal enable_mmu + + /* + * Calculate physical offset + * + * We can't re-use a value in phys_offset variable here as + * phys_offset is located in .bss and this section isn't + * 1:1 mapped and an access to it will cause MMU fault + */ + li t0, XEN_VIRT_START + la t1, start + sub t1, t1, t0 + + /* Calculate proper VA after jump from 1:1 mapping */ + la t0, .L_primary_switched + sub t0, t0, t1 + + /* Jump from 1:1 mapping world */ + jr t0 + +.L_primary_switched: + /* + * cpu0_boot_stack address is 1:1 mapping related so it should be + * recalculated after jump from 1:1 mapping world as 1:1 mapping + * will be removed soon in start_xen(). + */ + jal reset_stack + tail start_xen ENTRY(reset_stack)
xen/arch/riscv/setup.c +2 -12
@@ -11,20 +11,10 @@ unsigned char __initdata cpu0_boot_stack[STACK_SIZE]void __init noreturnstart_xen (unsigned long bootcpu_id,paddr_t dtb_addr) { -early_printk ("Hello from C env\n" ); - -setup_initial_pagetables (); - -enable_mmu (); - -for ( ;; ) - asmvolatile ("wfi" ); +remove_identity_mapping (); -unreachable (); -} +early_printk ("Hello from C env\n" ); -void __init noreturncont_after_mmu_is_enabled (void ) -{early_printk ("All set up\n" );for ( ;; )
Signature · 1 line
2.40.1
Quoted 4 lines · Oleksii Kurochko
The way how switch to virtual address was implemented in the commit
e66003e7be("xen/riscv: introducesetup_initial_pages") wasn't safe enough so identity mapping was introduced and used.
I don't think this is sufficient as a description. You want to make clear what the "not safe enough" is, and you also want to go into more detail as to the solution chosen. I'm particularly puzzled that you map just two singular pages ...
xen/arch/riscv/mm.c static unsigned long phys_offset; · line 46
Quoted 9 lines
xen/arch/riscv/mm.c +3
-1@@ -35,8 +40,10 @@ static unsigned long phys_offset;* * It might be needed one more page table in case when Xen load address * isn't 2 MB aligned. +* +* 3 additional page tables are needed for identity mapping. */ -#define PGTBL_INITIAL_COUNT ((CONFIG_PAGING_LEVELS - 1) + 1) +#define PGTBL_INITIAL_COUNT ((CONFIG_PAGING_LEVELS - 1) + 1 + 3)
What is this 3 coming from? It feels like the value should (again) somehow depend on CONFIG_PAGING_LEVELS.
xen/arch/riscv/mm.c static void __init setup_initial_mapping(struct mmu_desc *mmu_desc, · line 119
Quoted 5 lines
xen/arch/riscv/mm.c +1
-0@@ -108,16 +116,18 @@ static void __init setup_initial_mapping(struct mmu_desc *mmu_desc, {unsigned long paddr = (page_addr - map_start) + pa_start;unsigned int permissions =PTE_LEAF_DEFAULT ; + unsigned long addr = (is_identity_mapping) ?
Nit: No need for parentheses here.
xen/arch/riscv/mm.c static void __init setup_initial_mapping(struct mmu_desc *mmu_desc, · line 120
Quoted 1 line
xen/arch/riscv/mm.c +1
-0+ page_addr :LINK_TO_LOAD (page_addr);
As a remark, while we want binary operators at the end of lines when wrapping, we usually do things differently for the ternary operator:
Either
unsigned long addr = is_identity_mapping
? page_addr : LINK_TO_LOAD(page_addr);
or
unsigned long addr = is_identity_mapping
? page_addr
: LINK_TO_LOAD(page_addr);
.
Quoted 6 lines
xen/arch/riscv/mm.c +2
-0@@ -232,22 +242,27 @@ void __init setup_initial_pagetables(void) linker_start, linker_end, load_start); + +if ( linker_start == load_start )
xen/arch/riscv/mm.c void __init setup_initial_pagetables(void) · line 256
Quoted 10 lines
xen/arch/riscv/mm.c +10
-0+return ; + +setup_initial_mapping (&mmu_desc, + load_start, + load_start +PAGE_SIZE , + load_start); + + setup_initial_mapping(&mmu_desc, + (unsigned long)cpu0_boot_stack, + (unsigned long)cpu0_boot_stack + PAGE_SIZE,
Shouldn't this be STACK_SIZE (and then also be prepared for STACK_SIZE > PAGE_SIZE)?
xen/arch/riscv/mm.c void __init setup_initial_pagetables(void) · line 263
Quoted 8 lines
xen/arch/riscv/mm.c +5
-1+ (unsigned long)cpu0_boot_stack); } -void __init noreturn noinline enable_mmu() +/* +* enable_mmu() can't be __init because __init section isn't part of identity +* mapping so it will cause an issue after MMU will be enabled. +*/
As hinted at above already - perhaps the identity mapping wants to be larger, up to covering the entire Xen image? Since it's temporary only anyway, you could even consider using a large page (and RWX permission). You already require no overlap of link and load addresses, so at least small page mappings ought to be possible for the entire image.
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 277
Quoted 9 lines
xen/arch/riscv/mm.c +5
-0@@ -255,25 +270,41 @@ void __init noreturn noinline enable_mmu()csr_write (CSR_SATP ,PFN_DOWN ((unsigned long )stage1_pgtbl_root) |RV_STAGE1_MODE <<SATP_MODE_SHIFT ); +} + +void __init remove_identity_mapping (void ) +{ +int i, j;
Nit: unsigned int please.
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 279
Quoted 2 lines
xen/arch/riscv/mm.c +2
-0+pte_t *pgtbl; +unsigned int index, xen_index;
These would all probably better be declared in the narrowest possible scope.
Quoted 6 lines
xen/arch/riscv/mm.c +0
-5- asmvolatile (".p2align 2" ); - mmu_is_enabled:/* -* Stack should be re-inited as: -* 1. Right now an address of the stack is relative to load time -* addresses what will cause an issue in case of load start address
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 283
Quoted 10 lines
xen/arch/riscv/mm.c +2
-8-* isn't equal to linker start address. -* 2. Addresses in stack are all load time relative which can be an -* issue in case when load start address isn't equal to linker -* start address. -* -* We can't return to the caller because the stack was reseted -* and it may have stash some variable on the stack. -* Jump to a brand new function as the stack was reseted +* id_addrs should be in sync with id mapping in +* setup_initial_pagetables()
What is "id" meant to stand for here? Also if things need keeping in sync, then a similar comment should exist on the other side.
Quoted 12 lines
xen/arch/riscv/mm.c +8
-2*/ +unsigned long id_addrs[] = { +LINK_TO_LOAD (_start), +LINK_TO_LOAD (cpu0_boot_stack), + }; -switch_stack_and_jump ((unsigned long )cpu0_boot_stack +STACK_SIZE , - cont_after_mmu_is_enabled); + pgtbl = stage1_pgtbl_root; + +for ( j =0 ; j <ARRAY_SIZE (id_addrs); j++ ) + {
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 303
Quoted 10 lines
xen/arch/riscv/mm.c +10
-0+for ( pgtbl = stage1_pgtbl_root, i =CONFIG_PAGING_LEVELS -1 ; i >=0 ; i-- ) + { + index =pt_index (i, id_addrs[j]); + xen_index =pt_index (i,XEN_VIRT_START ); + +if ( index != xen_index ) + { + pgtbl[index].pte =0 ; +break ; + }
Up to here I understand index specifies a particular PTE within pgtbl[].
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 305
Quoted 1 line
xen/arch/riscv/mm.c +1
-0+ pgtbl = &pgtbl[index];
But then how can this be the continuation of the page table walk? Don't you need to read the address out of the PTE?
Quoted 10 lines
xen/arch/riscv/riscv64/head.S +4
-0@@ -31,6 +31,36 @@ ENTRY(start) jal calc_phys_offset + jal setup_initial_pagetables + + jal enable_mmu +
xen/arch/riscv/riscv64/head.S ENTRY(start) · line 47
Quoted 10 lines
xen/arch/riscv/riscv64/head.S +10
-0+ /* + * Calculate physical offset + * + * We can't re-use a value in phys_offset variable here as + * phys_offset is located in .bss and this section isn't + * 1:1 mapped and an access to it will cause MMU fault + */ + li t0, XEN_VIRT_START + la t1, start + sub t1, t1, t0
If I'm not mistaken this calculates start - XEN_VIRT_START, and ...
xen/arch/riscv/riscv64/head.S ENTRY(start) · line 51
Quoted 3 lines
xen/arch/riscv/riscv64/head.S +3
-0+ /* Calculate proper VA after jump from 1:1 mapping */ + la t0, .L_primary_switched + sub t0, t0, t1
... then this .L_primary_switched - (start - XEN_VIRT_START). Which can also be expressed as (.L_primary_switched - start) + XEN_VIRT_START, the first part of which gas ought to be able to resolve for you. But upon experimenting a little it looks like it can't. (I had thought of something along the lines of
li t0, .L_primary_switched - start
li t1, XEN_VIRT_START
add t0, t0, t1or even
li t1, XEN_VIRT_START
add t0, t1, %pcrel_lo(.L_primary_switched - start)
.)
Jan
Quoted 13 lines · Jan Beulich
xen/arch/riscv/mm.c void __init setup_initial_pagetables(void) · line 263
Quoted 5 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +4
-1-void __init noreturn noinline enable_mmu() +/* +* enable_mmu() can't be __init because __init section isn't part of identity +* mapping so it will cause an issue after MMU will be enabled. +*/ As hinted at above already - perhaps the identity mapping wants to be larger, up to covering the entire Xen image? Since it's temporary only anyway, you could even consider using a large page (and RWX permission). You already require no overlap of link and load addresses, so at least small page mappings ought to be possible for the entire image.
To expand on that: Assume a future change on this path results in a call to memcpy() or memset() being introduced by the compiler (and then let's further assume this only occurs for a specific compiler version). Right now such a case would be noticed simply because we don't build those library functions yet. But it'll likely be a perplexing crash once a full hypervisor can be built, the more that exception handlers also aren't mapped.
Quoted 16 lines
Quoted 5 lines
-
mmu_is_enabled:/*xen/arch/riscv/mm.c +0
-3-* Stack should be re-inited as: -* 1. Right now an address of the stack is relative to load time -* addresses what will cause an issue in case of load start address xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 283
Quoted 10 lines
xen/arch/riscv/mm.c +2
-8-* isn't equal to linker start address. -* 2. Addresses in stack are all load time relative which can be an -* issue in case when load start address isn't equal to linker -* start address. -* -* We can't return to the caller because the stack was reseted -* and it may have stash some variable on the stack. -* Jump to a brand new function as the stack was reseted +* id_addrs should be in sync with id mapping in +* setup_initial_pagetables()
Quoted 2 lines
What is "id" meant to stand for here? Also if things need keeping in sync, then a similar comment should exist on the other side.
I guess it's meant to stand for "identity mapping", but the common use of "id" makes we wonder if the variable wouldn't better be ident_addrs[].
Jan
Quoted 18 lines · Jan Beulich
Quoted 16 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +2
-1-void __init noreturn noinline enable_mmu() +/* +* enable_mmu() can't be __init because __init section isn't part of identity
+ * mapping so it will cause an issue after MMU will be enabled.
+ */As hinted at above already - perhaps the identity mapping wants to be
larger, up to covering the entire Xen image? Since it's temporary only anyway, you could even consider using a large page (and RWX permission). You already require no overlap of link and load addresses,
so at least small page mappings ought to be possible for the entire image.
Quoted 11 lines
To expand on that: Assume a future change on this path results in a call
tomemcpy()ormemset()being introduced by the compiler (and then let's
further assume this only occurs for a specific compiler version). Right
now such a case would be noticed simply because we don't build those library functions yet. But it'll likely be a perplexing crash once a full
hypervisor can be built, the more that exception handlers also aren't mapped.
It makes sense but for some reason it doesn't crash ( I have a bunch of patches to run dom0 ) but as I mentioned in the previous e-mail I agree that probably it would be better to map the whole image using 1 Gb page table for example.
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
-
mmu_is_enabled:
/*
- * Stack should be re-inited as:
- * 1. Right now an address of the stack is relative to load time
- * addresses what will cause an issue in case of load start address
- * isn't equal to linker start address.
- * 2. Addresses in stack are all load time relative which can be an
- * issue in case when load start address isn't equal to linker
- * start address.
- *
- * We can't return to the caller because the stack was reseted
- * and it may have stash some variable on the stack. - * Jump to a brand new function as the stack was reseted + *id_addrsshould be in sync with id mapping in + *setup_initial_pagetables()
Quoted 8 lines
Quoted 3 lines
What is "id" meant to stand for here? Also if things need keeping in
sync, then a similar comment should exist on the other side.I guess it's meant to stand for "identity mapping", but the common use
of "id" makes we wonder if the variable wouldn't better be ident_addrs[].
It would be better. but probably we will remove that variable if we agree to map the whole Xen instead of parts. So I'll wait for your response before starting to work on new patch series.
Thanks a lot.
~ Oleksii
Quoted 10 lines · Jan Beulich
Quoted 4 lines · Oleksii Kurochko
The way how switch to virtual address was implemented in the commit
e66003e7be("xen/riscv: introducesetup_initial_pages") wasn't safe enough so identity mapping was introduced and used.I don't think this is sufficient as a description. You want to make clear what the "not safe enough" is, and you also want to go into more detail as to the solution chosen. I'm particularly puzzled that you map just two singular pages ...
I'll update a descrption.
I map two pages as it likely to be enough to switch from 1:1 mapping world. My point is that we need 1:1 mapping only for few instructions which are located in start() [ in .text.header section ]:
li t0, XEN_VIRT_START
la t1, start
sub t1, t1, t0 /* Calculate proper VA after jump from 1:1 mapping */
la t0, .L_primary_switched
sub t0, t0, t1
/* Jump from 1:1 mapping world */
jr t0And it is needed to map 1:1 cpu0_boot_stack to not to fault when executing epilog of enable_mmu() function as it accesses a value on the stack:
ffffffffc00001b0: 6422 ld s0,8(sp)
ffffffffc00001b2: 0141 addi sp,sp,16
ffffffffc00001b4: 8082 retQuoted 14 lines
xen/arch/riscv/mm.c static unsigned long phys_offset; · line 46
Quoted 11 lines
(quoted diff) +0
-0@@ -35,8 +40,10 @@ static unsigned long phys_offset; * * It might be needed one more page table in case when Xen loadaddress
* isn't 2 MB aligned.
+ *
+ * 3 additional page tables are needed for identity mapping. */xen/arch/riscv/mm.c +1
-1-#define PGTBL_INITIAL_COUNT ((CONFIG_PAGING_LEVELS - 1) + 1) +#define PGTBL_INITIAL_COUNT ((CONFIG_PAGING_LEVELS - 1) + 1 + 3)What is this 3 coming from? It feels like the value should (again) somehow depend on
CONFIG_PAGING_LEVELS.
3 is too much in the current case. It should be 2 more.
The linker address is 0xFFFFFFFC00000000 thereby mapping the linker to load addresses
we need 3 pages ( with the condition that the size of Xen won't be larger than 2 MB ) and 1 page if the case when Xen load address isn't 2MV aligned.
We need 2 more page tables because Xen is loaded to address 0x80200000 by OpenSBI and the load address isn't equal to the linker address so we need additional 2 pages as the L2 table we already allocated when mapping the linker to load addresses.
And it looks like 2 is enough for Sv48, Sv57 as L4, L3 and L2 pagetables will be already allocated during mapping the linker to load addresses.
And it only will be too much for Sv32 ( which has only L1, L0 in general ) but I am not sure that anyone cares about Sv32.
Quoted 10 lines
Quoted 8 lines
(quoted diff) +0
-0@@ -108,16 +116,18 @@ static void __init setup_initial_mapping(struct mmu_desc *mmu_desc, {unsigned long paddr = (
page_addr-map_start) +pa_start;
unsigned int permissions =PTE_LEAF_DEFAULT; + unsigned long addr = (is_identity_mapping) ?Nit: No need for parentheses here.
Thanks.
Quoted 12 lines
Quoted 3 lines
+
page_addr:LINK_TO_LOAD(page_addr);As a remark, while we want binary operators at the end of lines when wrapping, we usually do things differently for the ternary operator:
Eitherunsigned long addr =
is_identity_mapping?page_addr:LINK_TO_LOAD(page_addr);
Quoted 5 lines
or
unsigned long addr =
is_identity_mapping?page_addr
:LINK_TO_LOAD(page_addr);
It looks better. Thanks.
Quoted 15 lines
.
Quoted 12 lines
xen/arch/riscv/mm.c +8
-0@@ -232,22 +242,27 @@ void __init setup_initial_pagetables(void) linker_start, linker_end, load_start); + + if ( linker_start == load_start ) + return; + + setup_initial_mapping(&mmu_desc, + load_start, + load_start + PAGE_SIZE, + load_start);
Quoted 8 lines
xen/arch/riscv/mm.c void __init setup_initial_pagetables(void)
Quoted 5 lines
xen/arch/riscv/mm.c +4
-0+ + setup_initial_mapping(&mmu_desc, + (unsigned long)cpu0_boot_stack, + (unsigned long)cpu0_boot_stack + PAGE_SIZE,Shouldn't this be
STACK_SIZE(and then also be prepared forSTACK_SIZE>PAGE_SIZE)?
Yes, it will be better to use STACK_SIZE but for 1:1 mapping it will be enough PAGE_SIZE as I mentioned above this only need for epilogue of enable_mmu() function.
Quoted 11 lines
Quoted 10 lines
xen/arch/riscv/mm.c +1
-0+ (unsigned long)cpu0_boot_stack); }xen/arch/riscv/mm.c +2
-1-void __init noreturn noinline enable_mmu() +/* +* enable_mmu() can't be __init because __init section isn't part of identity
+ * mapping so it will cause an issue after MMU will be enabled.
+ */
Quoted 7 lines
As hinted at above already - perhaps the identity mapping wants to be larger, up to covering the entire Xen image? Since it's temporary only anyway, you could even consider using a large page (and RWX permission). You already require no overlap of link and load addresses,
so at least small page mappings ought to be possible for the entire image.
It makes sense and started to thought about that after I applied the patch for Dom0 running... I think we can map 1 GB page which will cover the whole Xen image. Also in that case we haven't to allocate 2 more page tables.
Quoted 12 lines
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 277
Quoted 10 lines
xen/arch/riscv/mm.c +0
-0@@ -255,25 +270,41 @@ void __init noreturn noinline enable_mmu() csr_write(CSR_SATP,
PFN_DOWN((unsigned long)stage1_pgtbl_root)|RV_STAGE1_MODE<<SATP_MODE_SHIFT);xen/arch/riscv/mm.c +5
-0+} + +void __init remove_identity_mapping (void ) +{ + int i, j;Nit: unsigned int please.
Thanks.
Quoted 6 lines
Quoted 3 lines
+
pte_t*pgtbl;
+ unsigned int index,xen_index;These would all probably better be declared in the narrowest possible scope.
Sure.
Quoted 22 lines
Quoted 13 lines
- asm volatile ( ".p2align 2" );
-mmu_is_enabled:
/*
- * Stack should be re-inited as:
- * 1. Right now an address of the stack is relative to load time
- * addresses what will cause an issue in case of load start address
- * isn't equal to linker start address.
- * 2. Addresses in stack are all load time relative which can be an
- * issue in case when load start address isn't equal toQuoted 8 lines
linker
- * start address.
- *
- * We can't return to the caller because the stack was reseted - * and it may have stash some variable on the stack. - * Jump to a brand new function as the stack was reseted + *id_addrsshould be in sync with id mapping in
+ *setup_initial_pagetables()
Quoted 2 lines
What is "id" meant to stand for here? Also if things need keeping in sync, then a similar comment should exist on the other side.
"id" here mean identity.
Quoted 29 lines
Quoted 15 lines
*/
+ unsigned long id_addrs[] = {
+LINK_TO_LOAD(_start), +LINK_TO_LOAD(cpu0_boot_stack), + };
- switch_stack_and_jump((unsigned long)cpu0_boot_stack +STACK_SIZE,
-cont_after_mmu_is_enabled); + pgtbl =stage1_pgtbl_root;xen/arch/riscv/mm.c +4
-0+ + for ( j = 0; j < ARRAY_SIZE(id_addrs); j++ ) + { + for ( pgtbl = stage1_pgtbl_root, i = CONFIG_PAGING_LEVELSxen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 303
Quoted 10 lines
xen/arch/riscv/mm.c +9
-1- 1; i >= 0; i-- ) + { + index = pt_index(i, id_addrs[j]); + xen_index = pt_index(i, XEN_VIRT_START); + + if ( index != xen_index ) + { + pgtbl[index].pte = 0; + break; + }Up to here I understand index specifies a particular PTE within pgtbl[].
Quoted 5 lines
Quoted 1 line
+ pgtbl = &pgtbl[index];
But then how can this be the continuation of the page table walk? Don't
you need to read the address out of the PTE?
You are right. it should be:
pgtbl = (pte_t *)pte_to_paddr(pgtbl[index]);
Quoted 39 lines
Quoted 11 lines
xen/arch/riscv/riscv64/head.S +0
-0@@ -31,6 +31,36 @@ ENTRY(start)
jalcalc_phys_offset
+ jalsetup_initial_pagetablesxen/arch/riscv/riscv64/head.S +3
-0+ + jal enable_mmu +xen/arch/riscv/riscv64/head.S ENTRY(start) · line 47
Quoted 10 lines
xen/arch/riscv/riscv64/head.S +10
-0+ /* + * Calculate physical offset + * + * We can't re-use a value in phys_offset variable here as + * phys_offset is located in .bss and this section isn't + * 1:1 mapped and an access to it will cause MMU fault + */ + li t0, XEN_VIRT_START + la t1, start + sub t1, t1, t0If I'm not mistaken this calculates start -
XEN_VIRT_START, and ...Quoted 3 lines
+ /* Calculate proper VA after jump from 1:1 mapping */ + la t0, .L_primary_switched
+ sub t0, t0, t1... then this .L_primary_switched - (start -
XEN_VIRT_START). Which can
also be expressed as (.L_primary_switched - start) +XEN_VIRT_START, the first part of which gas ought to be able to resolve for you. But upon experimenting a little it looks like it can't. (I had thought of something along the lines ofli t0, .L_primary_switched - start
li t1,XEN_VIRT_START
add t0, t0, t1
Quoted 6 lines
or even
li t1,
XEN_VIRT_START
add t0, t1, %pcrel_lo(.L_primary_switched - start).)
Calculation of ".L_primary_switched - start" might be an issue for gcc. I tried to do something similar and recieved or "Error: can't resolve .L_primary_switched - start" or "Error: illegal operands `li t0,.L_primary_switched-start'"
~ Oleksii
Quoted 7 lines · Oleksii
Quoted 6 lines · Jan Beulich
Quoted 4 lines · Oleksii Kurochko
The way how switch to virtual address was implemented in the commit
e66003e7be("xen/riscv: introducesetup_initial_pages") wasn't safe enough so identity mapping was introduced and used.
Quoted 7 lines
Quoted 4 lines
I don't think this is sufficient as a description. You want to make clear what the "not safe enough" is, and you also want to go into more detail as to the solution chosen. I'm particularly puzzled that you map just two singular pages ...
I'll update a descrption.
I map two pages as it likely to be enough to switch from 1:1 mapping
I dislike your use of "likely" here: Unless this code is meant to be redone anyway, it should just work. Everywhere.
Quoted 43 lines
world. My point is that we need 1:1 mapping only for few instructions which are located in
start()[ in .text.header section ]:li t0, XEN_VIRT_START la t1, start sub t1, t1, t0/* Calculate proper VA after jump from 1:1 mapping */ la t0, .L_primary_switched sub t0, t0, t1 /* Jump from 1:1 mapping world */ jr t0And it is needed to map 1:1
cpu0_boot_stackto not to fault when executing epilog ofenable_mmu()function as it accesses a value on the stack:ffffffffc00001b0: 6422 ld s0,8(sp) ffffffffc00001b2: 0141 addi sp,sp,16 ffffffffc00001b4: 8082 retQuoted 14 lines
xen/arch/riscv/mm.c static unsigned long phys_offset; · line 46
Quoted 11 lines
(quoted diff) +0
-0@@ -35,8 +40,10 @@ static unsigned long phys_offset; * * It might be needed one more page table in case when Xen loadaddress
* isn't 2 MB aligned.
+ *
+ * 3 additional page tables are needed for identity mapping. */xen/arch/riscv/mm.c +1
-1-#define PGTBL_INITIAL_COUNT ((CONFIG_PAGING_LEVELS - 1) + 1) +#define PGTBL_INITIAL_COUNT ((CONFIG_PAGING_LEVELS - 1) + 1 + 3)What is this 3 coming from? It feels like the value should (again) somehow depend on
CONFIG_PAGING_LEVELS.3 is too much in the current case. It should be 2 more.
The linker address is
0xFFFFFFFC00000000thereby mapping the linker to load addresses
we need 3 pages ( with the condition that the size of Xen won't be larger than 2 MB ) and 1 page if the case when Xen load address isn't 2MV aligned.
Quoted 8 lines
We need 2 more page tables because Xen is loaded to address
0x80200000by OpenSBI and the load address isn't equal to the linker address so we need additional 2 pages as the L2 table we already allocated when mapping the linker to load addresses.And it looks like 2 is enough for Sv48, Sv57 as L4, L3 and L2 pagetables will be already allocated during mapping the linker to load addresses.
I agree the root table will be there. But depending on load address, I don't think you can rely on any other tables to be re-usable from the Xen mappings you already have. So my gut feeling would be
#define PGTBL_INITIAL_COUNT ((CONFIG_PAGING_LEVELS - 1) * 2 + 1)
Quoted 20 lines
Quoted 12 lines
Quoted 12 lines
xen/arch/riscv/mm.c +8
-0@@ -232,22 +242,27 @@ void __init setup_initial_pagetables(void) linker_start, linker_end, load_start); + + if ( linker_start == load_start ) + return; + + setup_initial_mapping(&mmu_desc, + load_start, + load_start + PAGE_SIZE, + load_start);Quoted 8 lines
xen/arch/riscv/mm.c void __init setup_initial_pagetables(void)
Quoted 5 lines
xen/arch/riscv/mm.c +4
-0+ + setup_initial_mapping(&mmu_desc, + (unsigned long)cpu0_boot_stack, + (unsigned long)cpu0_boot_stack + PAGE_SIZE,Shouldn't this be
STACK_SIZE(and then also be prepared forSTACK_SIZE>PAGE_SIZE)?
Quoted 3 lines
Yes, it will be better to use
STACK_SIZEbut for 1:1 mapping it will be enoughPAGE_SIZEas I mentioned above this only need for epilogue ofenable_mmu()function.
But then it should still be the correct page of the stack that you map (the top one aiui).
Quoted 14 lines
Quoted 7 lines
Quoted 6 lines
xen/arch/riscv/mm.c +2
-1-void __init noreturn noinline enable_mmu() +/* +* enable_mmu() can't be __init because __init section isn't part of identity
+ * mapping so it will cause an issue after MMU will be enabled.
+ */Quoted 7 lines
As hinted at above already - perhaps the identity mapping wants to be larger, up to covering the entire Xen image? Since it's temporary only anyway, you could even consider using a large page (and RWX permission). You already require no overlap of link and load addresses,
so at least small page mappings ought to be possible for the entire image.
Quoted 4 lines
It makes sense and started to thought about that after I applied the patch for Dom0 running... I think we can map 1 GB page which will cover the whole Xen image. Also in that case we haven't to allocate 2 more page tables.
But you then need to be careful about not mapping space accesses to which may have side effects (i.e. non-RAM, which might be MMIO or holes). Otherwise speculative execution might cause surprises, unless such is excluded by other guarantees (of which I don't know).
Quoted 40 lines
Quoted 40 lines
Quoted 10 lines
xen/arch/riscv/riscv64/head.S +0
-0@@ -31,6 +31,36 @@ ENTRY(start)
jalcalc_phys_offset
+ jalsetup_initial_pagetablesxen/arch/riscv/riscv64/head.S +3
-0+ + jal enable_mmu +xen/arch/riscv/riscv64/head.S ENTRY(start) · line 47
Quoted 10 lines
xen/arch/riscv/riscv64/head.S +10
-0+ /* + * Calculate physical offset + * + * We can't re-use a value in phys_offset variable here as + * phys_offset is located in .bss and this section isn't + * 1:1 mapped and an access to it will cause MMU fault + */ + li t0, XEN_VIRT_START + la t1, start + sub t1, t1, t0If I'm not mistaken this calculates start -
XEN_VIRT_START, and ...Quoted 3 lines
+ /* Calculate proper VA after jump from 1:1 mapping */ + la t0, .L_primary_switched
+ sub t0, t0, t1... then this .L_primary_switched - (start -
XEN_VIRT_START). Which can
also be expressed as (.L_primary_switched - start) +XEN_VIRT_START, the first part of which gas ought to be able to resolve for you. But upon experimenting a little it looks like it can't. (I had thought of something along the lines ofli t0, .L_primary_switched - start
li t1,XEN_VIRT_START
add t0, t0, t1or even
Quoted 8 lines
Quoted 4 lines
li t1,
XEN_VIRT_START
add t0, t1, %pcrel_lo(.L_primary_switched - start).)
Calculation of ".L_primary_switched - start" might be an issue for gcc. I tried to do something similar and recieved or "Error: can't resolve .L_primary_switched - start" or "Error: illegal operands `li t0,.L_primary_switched-start'"
Matches the results of my experiments. If I can find time, I may want to look into why exactly gas is rejecting such.
Jan
Quoted 9 lines
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 277
Quoted 5 lines
xen/arch/riscv/mm.c +5
-0+} + +void __init remove_identity_mapping (void ) +{ + int i, j;Nit: unsigned int please.
It should be int in the current case because of the 'for' exit condition:
for ( pgtbl = stage1_pgtbl_root, i = CONFIG_PAGING_LEVELS - 1; i
Quoted 1 line
= 0; i-- )
Should exit condition be re-writen?
~ Oleksii
Quoted 9 lines · Oleksii
Quoted 9 lines
xen/arch/riscv/mm.c void __init noreturn noinline enable_mmu() · line 277
Quoted 5 lines
xen/arch/riscv/mm.c +5
-0+} + +void __init remove_identity_mapping (void ) +{ + int i, j;Nit: unsigned int please.
Quoted 6 lines
It should be int in the current case because of the 'for' exit condition:
for ( pgtbl =stage1_pgtbl_root, i =CONFIG_PAGING_LEVELS- 1; iQuoted 1 line
= 0; i-- )
Should exit condition be re-writen?
Since it easily can be, I think that would be preferable. But in a case like this, if you think it's better the way you have it, so be it (until someone comes along and changes it).
Jan
PATCH 6/8 xen/riscv: add SPDX tags
In reply to Oleksii Kurochko--- xen/arch/riscv/include/asm/config.h | 2 ++ xen/arch/riscv/include/asm/current.h | 2 ++ xen/arch/riscv/include/asm/early_printk.h | 2 ++ xen/arch/riscv/include/asm/mm.h | 2 ++ xen/arch/riscv/include/asm/page-bits.h | 2 ++ xen/arch/riscv/include/asm/page.h | 2 ++ xen/arch/riscv/include/asm/traps.h | 2 ++ xen/arch/riscv/include/asm/types.h | 2 ++ xen/arch/riscv/mm.c | 2 ++ xen/arch/riscv/setup.c | 2 ++ 10 files changed, 20 insertions(+)
xen/arch/riscv/include/asm/config.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef __RISCV_CONFIG_H__#define __RISCV_CONFIG_H__
xen/arch/riscv/include/asm/current.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef __ASM_CURRENT_H#define __ASM_CURRENT_H
xen/arch/riscv/include/asm/early_printk.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef __EARLY_PRINTK_H__#define __EARLY_PRINTK_H__
xen/arch/riscv/include/asm/mm.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _ASM_RISCV_MM_H#define _ASM_RISCV_MM_H
xen/arch/riscv/include/asm/page-bits.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef __RISCV_PAGE_BITS_H__#define __RISCV_PAGE_BITS_H__
xen/arch/riscv/include/asm/page.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _ASM_RISCV_PAGE_H#define _ASM_RISCV_PAGE_H
xen/arch/riscv/include/asm/traps.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef __ASM_TRAPS_H__#define __ASM_TRAPS_H__
xen/arch/riscv/include/asm/types.h +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef __RISCV_TYPES_H__#define __RISCV_TYPES_H__
xen/arch/riscv/mm.c +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#include <xen/compiler.h> #include <xen/init.h> #include <xen/kernel.h>
xen/arch/riscv/setup.c +2 -0
@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#include <xen/compile.h> #include <xen/init.h>
Signature · 1 line
2.40.1
Re: [PATCH v1 6/8] xen/riscv: add SPDX tags
Hi Oleksii,
Quoted 19 lines · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>--- xen/arch/riscv/include/asm/config.h | 2 ++ xen/arch/riscv/include/asm/current.h | 2 ++ xen/arch/riscv/include/asm/early_printk.h | 2 ++ xen/arch/riscv/include/asm/mm.h | 2 ++ xen/arch/riscv/include/asm/page-bits.h | 2 ++ xen/arch/riscv/include/asm/page.h | 2 ++ xen/arch/riscv/include/asm/traps.h | 2 ++ xen/arch/riscv/include/asm/types.h | 2 ++ xen/arch/riscv/mm.c | 2 ++ xen/arch/riscv/setup.c | 2 ++ 10 files changed, 20 insertions(+)xen/arch/riscv/include/asm/config.h +0
-0
xen/arch/riscv/include/asm/config.h line 1
Quoted 1 line
xen/arch/riscv/include/asm/config.h +1
-0@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */
The default license for Xen is GPL-2.0-only. So any particular reason to diverge?
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v1 6/8] xen/riscv: add SPDX tags
Hi Julien,
Quoted 17 lines · Julien Grall
Hi Oleksii,
Quoted 14 lines · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>---
xen/arch/riscv/include/asm/config.h | 2 ++ xen/arch/riscv/include/asm/current.h | 2 ++xen/arch/riscv/include/asm/early_printk.h| 2 ++ xen/arch/riscv/include/asm/mm.h | 2 ++ xen/arch/riscv/include/asm/page-bits.h | 2 ++ xen/arch/riscv/include/asm/page.h | 2 ++ xen/arch/riscv/include/asm/traps.h | 2 ++ xen/arch/riscv/include/asm/types.h | 2 ++ xen/arch/riscv/mm.c | 2 ++ xen/arch/riscv/setup.c | 2 ++10 files changed, 20 insertions(+)
Quoted 11 lines
xen/arch/riscv/include/asm/config.h line 1
Quoted 7 lines
xen/arch/riscv/include/asm/config.h +1
-0@@ -1,3 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ The default license for Xen is GPL-2.0-only. So any particular reason to
diverge?
Thanks for a remark.
There is no any particular reason for that. I copied it from on of a file from RISC-V arch directory.
I'll change it to GPL-2.0-only in the next version of the patch series.
~ Oleksii
PATCH 7/8 xen/riscv: add __ASSEMBLY__ guards
In reply to Oleksii Kurochko--- xen/arch/riscv/include/asm/page.h | 4 ++++ 1 file changed, 4 insertions(+)
xen/arch/riscv/include/asm/page.h +4 -0
@@ -3,6 +3,8 @@#ifndef _ASM_RISCV_PAGE_H#define _ASM_RISCV_PAGE_H +#ifndef __ASSEMBLY__ +#include <xen/const.h> #include <xen/types.h>
@@ -60,4 +62,6 @@ static inline bool pte_is_valid(pte_t p)return p.pte &PTE_VALID ; } +#endif /* __ASSEMBLY__ */ +#endif /* _ASM_RISCV_PAGE_H */
Signature · 1 line
2.40.1
Re: [PATCH v1 7/8] xen/riscv: add __ASSEMBLY__ guards
Quoted 2 lines · Oleksii Kurochko
Signed-off-by Oleksii Kurochko <[email protected]>
Alistair
Quoted 28 lines
--- xen/arch/riscv/include/asm/page.h | 4 ++++ 1 file changed, 4 insertions(+)xen/arch/riscv/include/asm/page.h +4
-1@@ -3,6 +3,8 @@#ifndef _ASM_RISCV_PAGE_H#define _ASM_RISCV_PAGE_H +#ifndef __ASSEMBLY__ +#include <xen/const.h> #include <xen/types.h> @@ -60,4 +62,6 @@ static inline bool pte_is_valid(pte_t p)return p.pte &PTE_VALID ; } +#endif /* __ASSEMBLY__ */ +#endif /* _ASM_RISCV_PAGE_H */ --2.40.1
PATCH 8/8 xen/riscv: move extern of cpu0_boot_stack to header
In reply to Oleksii Kurochko--- xen/arch/riscv/include/asm/mm.h | 2 ++ xen/arch/riscv/mm.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
xen/arch/riscv/include/asm/mm.h +2 -0
@@ -8,6 +8,8 @@#define pfn_to_paddr (pfn) ((paddr_t)(pfn) << PAGE_SHIFT)#define paddr_to_pfn (pa) ((unsigned long)((pa) >> PAGE_SHIFT)) +extern unsigned char cpu0_boot_stack[]; +void setup_initial_pagetables (void );void enable_mmu (void );
xen/arch/riscv/mm.c +0 -2
@@ -19,8 +19,6 @@ struct mmu_desc {pte_t *pgtbl_base; }; -extern unsigned char cpu0_boot_stack[STACK_SIZE ]; -static unsigned long phys_offset;#define LOAD_TO_LINK (addr) ((unsigned long)(addr) - phys_offset)
Signature · 1 line
2.40.1
Quoted 6 lines · Oleksii Kurochko
The patch series introduces things necessary to implement identity mapping:
1. Make identity mapping for_startand stack.
2. Enable MMU.
3. Jump to the virtual address world
4. Remove identity mapping for_startand stack.
Quoted 8 lines
Also current patch series introduces the calculation of physical offset before MMU is enabled as access to physical offset will be calculated wrong after MMU will be enabled because access to
phys_offvariable is PC-relative and in the case when linker address != load address, it will cause MMU fault.One more thing that was done is:
* Added SPDX tags.
* Added__ASSEMBLY__guards.
These are are, aiui, a response to a comment from Andrew. Hence I think this wants expressing by a {Requested,Suggested,Reported}-by: tag in the respective patch.
Jan
Quoted 29 lines
* move extern of
cpu0_boot_stackto a header.The reason for this patch series can be found here:
https://lore.kernel.org/xen-devel/[email protected]/Oleksii Kurochko (8):
xen/riscv: make sure that identity mapping isn't bigger then page size xen/riscv: add .sbss section to .bss
xen/riscv: introducereset_stack()function
xen/riscv: introduce function for physical offset calculation xen/riscv: introduce identity mapping
xen/riscv: add SPDX tags
xen/riscv: add__ASSEMBLY__guards
xen/riscv: move extern ofcpu0_boot_stackto headerxen/arch/riscv/include/asm/config.h | 2 + xen/arch/riscv/include/asm/current.h | 2 + xen/arch/riscv/include/asm/early_printk.h | 2 + xen/arch/riscv/include/asm/mm.h | 9 +- xen/arch/riscv/include/asm/page-bits.h | 2 + xen/arch/riscv/include/asm/page.h | 6 ++ xen/arch/riscv/include/asm/traps.h | 2 + xen/arch/riscv/include/asm/types.h | 2 + xen/arch/riscv/mm.c | 119 +++++++++++++++------- xen/arch/riscv/riscv64/head.S | 40 +++++++- xen/arch/riscv/setup.c | 16 +-- xen/arch/riscv/xen.lds.S | 11 +- 12 files changed, 160 insertions(+), 53 deletions(-)
Quoted 21 lines · Jan Beulich
Quoted 16 lines · Oleksii Kurochko
The patch series introduces things necessary to implement identity mapping:
1. Make identity mapping for_startand stack.
2. Enable MMU.
3. Jump to the virtual address world
4. Remove identity mapping for_startand stack.Also current patch series introduces the calculation of physical offset before
MMU is enabled as access to physical offset will be calculated wrong after
MMU will be enabled because access tophys_offvariable is PC- relative and
in the case when linker address != load address, it will cause MMU fault.Quoted 3 lines
One more thing that was done is:
* Added SPDX tags.
* Added__ASSEMBLY__guards.
Quoted 5 lines
These are are, aiui, a response to a comment from Andrew. Hence I think
this wants expressing by a {Requested,Suggested,Reported}-by: tag in the
respective patch.
Thanks for a remark.
~ Oleksii
Quoted 32 lines
Quoted 31 lines
* move extern of
cpu0_boot_stackto a header.The reason for this patch series can be found here:
https://lore.kernel.org/xen-devel/[email protected]/Oleksii Kurochko (8):
xen/riscv: make sure that identity mapping isn't bigger then page size
xen/riscv: add .sbss section to .bss
xen/riscv: introducereset_stack()function
xen/riscv: introduce function for physical offset calculation xen/riscv: introduce identity mapping
xen/riscv: add SPDX tags
xen/riscv: add__ASSEMBLY__guards
xen/riscv: move extern ofcpu0_boot_stackto headerxen/arch/riscv/include/asm/config.h | 2 + xen/arch/riscv/include/asm/current.h | 2 +
xen/arch/riscv/include/asm/early_printk.h| 2 + xen/arch/riscv/include/asm/mm.h | 9 +- xen/arch/riscv/include/asm/page-bits.h | 2 + xen/arch/riscv/include/asm/page.h | 6 ++ xen/arch/riscv/include/asm/traps.h | 2 + xen/arch/riscv/include/asm/types.h | 2 + xen/arch/riscv/mm.c | 119 +++++++++++++++---
----
xen/arch/riscv/riscv64/head.S | 40 +++++++- xen/arch/riscv/setup.c | 16 +-- xen/arch/riscv/xen.lds.S | 11 +-12 files changed, 160 insertions(+), 53 deletions(-)