Skip to contentSkip to conversation

Add minimal RISC-V Xen build and build testing

PATCH In review Oleksii Kurochko 3 participants 8 messages Jan 5, 2023 RISC-V

lore
Share & tools
lore
#0 lore raw

v4 cover Add minimal RISC-V Xen build and build testing

The patch series introduces the following:
- provide a minimal amount of changes to add initial RISC-V support to make Xen binary buildable and runnable for RISC-V architecture which can be used for future development and testing.
- add RISC-V 64 cross-compile build jobs to check if any new changes break RISC-V build.

Changes in V4:
- Rebase on mainline 'staging'.
- Code style fixes and commit message updates.
- Minor changes of riscv/Makefile and build.yaml

Changes in V3:
- Remove include of <asm/config.h> from head.S.

Changes in V2:
- Remove the patch "automation: add cross-compiler support
for the build script" because it was reworked as a part of the patch series "CI: Fixes/cleanup in preparation for RISCV".
- Remove the patch "automation: add python3 package for riscv64.dockerfile" because it is not necessary for RISCV Xen binary build now.
- Rework the patch "arch/riscv: initial RISC-V support to build/run minimal Xen" according to the comments about v1 of the patch series.
- Add HYPERVISOR_ONLY to RISCV jobs in build.yaml after rebasing on "CI: Fixes/cleanup in preparation for RISCV" patch series.

Oleksii Kurochko (2):
arch/riscv: initial RISC-V support to build/run minimal Xen
automation: add RISC-V 64 cross-build tests for Xen

 automation/gitlab-ci/build.yaml     |  56 ++++++++++
 xen/arch/riscv/Makefile             |  14 +++
 xen/arch/riscv/arch.mk              |   4 +
 xen/arch/riscv/include/asm/config.h |   9 +-
 xen/arch/riscv/riscv64/Makefile     |   2 +-
 xen/arch/riscv/riscv64/head.S       |   4 +-
 xen/arch/riscv/xen.lds.S            | 158 ++++++++++++++++++++++++++++
 7 files changed, 242 insertions(+), 5 deletions(-)
 create mode 100644 xen/arch/riscv/xen.lds.S
Signature · 1 line
2.38.1
#7 lore raw

PATCH v4 2/2 automation: add RISC-V 64 cross-build tests for Xen

Add build jobs to cross-compile Xen-only for RISC-V 64.

---
Changes in V4:
- Add RISCV RANDCONFIG jobs
- Remove unnecessary comments
---
Changes in V2:
- Add HYPERVISOR_ONLY to RISCV jobs because after rebase on top of the patch series "CI: Fixes/cleanup in preparation for RISCV" it is required to set HYPERVISOR_ONLY in build.yaml

---
 automation/gitlab-ci/build.yaml | 56 +++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
automation/gitlab-ci/build.yaml +56 -0
index 43dbef8aba..6784974619 100644
@@ -172,6 +172,33 @@
   variables:
     <<: *gcc
 
+.riscv64-cross-build-tmpl:
+  <<: *build
+  variables:
+    XEN_TARGET_ARCH: riscv64
+  tags:
+    - x86_64
+
+.riscv64-cross-build:
+  extends: .riscv64-cross-build-tmpl
+  variables:
+    debug: n
+
+.riscv64-cross-build-debug:
+  extends: .riscv64-cross-build-tmpl
+  variables:
+    debug: y
+
+.gcc-riscv64-cross-build:
+  extends: .riscv64-cross-build
+  variables:
+    <<: *gcc
+
+.gcc-riscv64-cross-build-debug:
+  extends: .riscv64-cross-build-debug
+  variables:
+    <<: *gcc
+
 # Jobs below this line
 
 archlinux-gcc:
@@ -619,6 +646,35 @@ alpine-3.12-gcc-debug-arm64-boot-cpupools:
     EXTRA_XEN_CONFIG: |
       CONFIG_BOOT_TIME_CPUPOOLS=y
 
+# RISC-V 64 cross-build
+riscv64-cross-gcc:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: archlinux:riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+riscv64-cross-gcc-debug:
+  extends: .gcc-riscv64-cross-build-debug
+  variables:
+    CONTAINER: archlinux:riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+riscv64-cross-gcc-randconfig:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: archlinux:riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    RANDCONFIG: y
+
+riscv64-cross-gcc-debug-randconfig:
+  extends: .gcc-riscv64-cross-build-debug
+  variables:
+    CONTAINER: archlinux:riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    RANDCONFIG: y
+
 ## Test artifacts common
 
 .test-jobs-artifact-common:
Signature · 1 line
2.38.1
#1 lore raw

PATCH v4 1/2 arch/riscv: initial RISC-V support to build/run minimal Xen

In reply to Oleksii Kurochko

The patch provides a minimal amount of changes to start build and run minimal Xen binary at GitLab CI&CD that will allow continuous checking of the build status of RISC-V Xen.

Except introduction of new files the following changes were done:
* Redefinition of ALIGN define from '.align 2' to '.align 4'. '.align 2' was incorrect choice done previously
* ALL_OBJ-y and ALL_LIBS-y were temporary overwritted to produce a minimal hypervisor image otherwise it will be required to push huge amount of headers and stubs for common, drivers, libs etc which aren't necessary for now.
* Section changed from .text to .text.header for start function to make it the first one executed.
* Rework riscv64/Makefile logic to rebase over changes since the first RISC-V commit.

RISC-V Xen can be built by the following instructions:

  $ CONTAINER=riscv64 ./automation/scripts/containerize \
       make XEN_TARGET_ARCH=riscv64 -C xen tiny64_defconfig
  $ CONTAINER=riscv64 ./automation/scripts/containerize \
       make XEN_TARGET_ARCH=riscv64 -C xen build

RISC-V Xen can be run as:

  $ qemu-system-riscv64 -M virt -smp 1 -nographic -m 2g \
       -kernel xen/xen

To run in debug mode should be done the following instructions:

 $ qemu-system-riscv64 -M virt -smp 1 -nographic -m 2g \
        -kernel xen/xen -s -S
 # In separate terminal:
 $ riscv64-buildroot-linux-gnu-gdb
 $ target remote :1234
 $ add-symbol-file <xen_src>/xen/xen-syms 0x80200000
 $ hb *0x80200000
 $ c # it should stop at instruction j 0x80200000 <start>

---
Changes in V4:
- Remove clean-files target from Makefile as ${TARGET-syms} was simplifed and there is no any sense for clean-files target for now.
- Update the commit message.
- Code style fixes.
---
Changes in V3:
- Remove include of <asm/config.h> from head.S
---
Changes in V2:
- Update commit message:
- Add explanation why ALIGN define was changed.
- Add explanation why section of 'start' function was changed.
- Rework xen.lds.S linker script. It is mostly based on ARM except ARM-specific sections which have been removed.
- Rework in riscv64/Makefile rule $(TARGET)-syms
- Remove asm/types.h header as after reworking of riscv64/Makefile it is not needed now.
- Remove unneeded define SYMBOLS_DUMMY_OBJ.

---
 xen/arch/riscv/Makefile             |  14 +++
 xen/arch/riscv/arch.mk              |   4 +
 xen/arch/riscv/include/asm/config.h |   9 +-
 xen/arch/riscv/riscv64/Makefile     |   2 +-
 xen/arch/riscv/riscv64/head.S       |   4 +-
 xen/arch/riscv/xen.lds.S            | 158 ++++++++++++++++++++++++++++
 6 files changed, 186 insertions(+), 5 deletions(-)
 create mode 100644 xen/arch/riscv/xen.lds.S
xen/arch/riscv/Makefile +14 -0
index 942e4ffbc1..248f2cbb3e 100644
@@ -1,2 +1,16 @@
+obj-$(CONFIG_RISCV_64) += riscv64/
+
+$(TARGET): $(TARGET)-syms
+	$(OBJCOPY) -O binary -S $< $@
+
+$(TARGET)-syms: $(objtree)/prelink.o $(obj)/xen.lds
+	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) -o $@
+	$(NM) -pa --format=sysv $(@D)/$(@F) \
+		| $(objtree)/tools/symbols --all-symbols --xensyms --sysv --sort \
+		>$(@D)/$(@F).map
+
+$(obj)/xen.lds: $(src)/xen.lds.S FORCE
+	$(call if_changed_dep,cpp_lds_S)
+
 .PHONY: include
 include:
xen/arch/riscv/arch.mk +4 -0
index ae8fe9dec7..012dc677c3 100644
@@ -11,3 +11,7 @@ riscv-march-$(CONFIG_RISCV_ISA_C)       := $(riscv-march-y)c
 # -mcmodel=medlow would force Xen into the lower half.
 
 CFLAGS += -march=$(riscv-march-y) -mstrict-align -mcmodel=medany
+
+# TODO: Drop override when more of the build is working
+override ALL_OBJS-y = arch/$(TARGET_ARCH)/built_in.o
+override ALL_LIBS-y =
xen/arch/riscv/include/asm/config.h +8 -1
index e2ae21de61..c6f9e230d7 100644
@@ -1,6 +1,9 @@
 #ifndef __RISCV_CONFIG_H__
 #define __RISCV_CONFIG_H__
 
+#include <xen/const.h>
+#include <xen/page-size.h>
+
 #if defined(CONFIG_RISCV_64)
 # define LONG_BYTEORDER 3
 # define ELFSIZE 64
@@ -28,7 +31,7 @@
 
 /* Linkage for RISCV */
 #ifdef __ASSEMBLY__
-#define ALIGN .align 2
+#define ALIGN .align 4
 
 #define ENTRY(name)                                \
   .globl name;                                     \
@@ -36,6 +39,10 @@
   name:
 #endif
 
+#define XEN_VIRT_START  _AT(UL, 0x00200000)
+
+#define SMP_CACHE_BYTES (1 << 6)
+
 #endif /* __RISCV_CONFIG_H__ */
 /*
  * Local variables:
xen/arch/riscv/riscv64/Makefile +1 -1
index 15a4a65f66..3340058c08 100644
@@ -1 +1 @@
-extra-y += head.o
+obj-y += head.o
xen/arch/riscv/riscv64/head.S +1 -3
index 0dbc27ba75..990edb70a0 100644
@@ -1,6 +1,4 @@
-#include <asm/config.h>
-
-        .text
+        .section .text.header, "ax", %progbits
 
 ENTRY(start)
         j  start
xen/arch/riscv/xen.lds.S new file +158 -0

158 lines not on this page · raw message

Signature · 1 line
2.38.1
#2 lore raw

Re: [PATCH v4 1/2] arch/riscv: initial RISC-V support to build/run minimal Xen

Quoted 9 lines · Oleksii Kurochko

To run in debug mode should be done the following instructions:

 $ qemu-system-riscv64 -M virt -smp 1 -nographic -m 2g \
        -kernel xen/xen -s -S
 # In separate terminal:
 $ riscv64-buildroot-linux-gnu-gdb
 $ target remote :1234
 $ add-symbol-file <xen_src>/xen/xen-syms 0x80200000
 $ hb *0x80200000
 $ c # it should stop at instruction j 0x80200000 <start>

This suggests to me that Xen is meant to run at VA 0x80200000, whereas ...

Quoted 22 lines
xen/arch/riscv/include/asm/config.h +4 -1
@@ -1,6 +1,9 @@
 #ifndef __RISCV_CONFIG_H__
 #define __RISCV_CONFIG_H__
 
+#include <xen/const.h>
+#include <xen/page-size.h>
+
 #if defined(CONFIG_RISCV_64)
 # define LONG_BYTEORDER 3
 # define ELFSIZE 64
@@ -28,7 +31,7 @@
 
 /* Linkage for RISCV */
 #ifdef __ASSEMBLY__
-#define ALIGN .align 2
+#define ALIGN .align 4
 
 #define ENTRY(name)                                \
   .globl name;                                     \

xen/arch/riscv/include/asm/config.h line 42

Quoted 4 lines
xen/arch/riscv/include/asm/config.h +1 -0
@@ -36,6 +39,10 @@
   name:
 #endif
 
+#define XEN_VIRT_START  _AT(UL, 0x00200000)

... here you specify a much lower address (and to be honest even 0x80200000 looks pretty low to me for 64-bit, and perhaps even for 32-bit). Could you clarify what the plans here are? Maybe this is merely a temporary thing, but not called out as such?

Jan

#3 lore raw

Re: [PATCH v4 1/2] arch/riscv: initial RISC-V support to build/run minimal Xen

Quoted 37 lines · Jan Beulich
Quoted 9 lines · Oleksii Kurochko

To run in debug mode should be done the following instructions:

 $ qemu-system-riscv64 -M virt -smp 1 -nographic -m 2g \
        -kernel xen/xen -s -S
 # In separate terminal:
 $ riscv64-buildroot-linux-gnu-gdb
 $ target remote :1234
 $ add-symbol-file <xen_src>/xen/xen-syms 0x80200000
 $ hb *0x80200000
 $ c # it should stop at instruction j 0x80200000 <start>

This suggests to me that Xen is meant to run at VA 0x80200000, whereas ...

Quoted 25 lines
xen/arch/riscv/include/asm/config.h +4 -1
@@ -1,6 +1,9 @@
 #ifndef __RISCV_CONFIG_H__
 #define __RISCV_CONFIG_H__
 
+#include <xen/const.h>
+#include <xen/page-size.h>
+
 #if defined(CONFIG_RISCV_64)
 # define LONG_BYTEORDER 3
 # define ELFSIZE 64
@@ -28,7 +31,7 @@
 
 /* Linkage for RISCV */
 #ifdef __ASSEMBLY__
-#define ALIGN .align 2
+#define ALIGN .align 4
 
 #define ENTRY(name)                                \
   .globl name;                                     \
@@ -36,6 +39,10 @@
   name:
 #endif
 
Quoted 5 lines

xen/arch/riscv/include/asm/config.h line 42

Quoted 1 line
xen/arch/riscv/include/asm/config.h +1 -0
+#define XEN_VIRT_START  _AT(UL, 0x00200000)

... here you specify a much lower address (and to be honest even 0x80200000 looks pretty low to me for 64-bit, and perhaps even for 32-bit). Could you clarify what the plans here are? Maybe this is merely a temporary thing, but not called out as such?

It's stale from v1 which had:

#define XEN_VIRT_START  0x80200000

But honestly, I don't think the qemu details in the commit message are useful.  This series is just about making "make build" work.

The next series (being worked on, but not posted yet) is only a few patches and gets a full Gitlab CI smoke test, at which point the smoke test shell script is the reference for how to invoke qemu.

I'm happy to R-by this series and drop that part of the commit message on commit.

~Andrew

author
#4 lore raw

Re: [PATCH v4 1/2] arch/riscv: initial RISC-V support to build/run minimal Xen

Quoted 63 lines · Andrew Cooper
Quoted 38 lines · Jan Beulich
Quoted 9 lines · Oleksii Kurochko

To run in debug mode should be done the following instructions:

 $ qemu-system-riscv64 -M virt -smp 1 -nographic -m 2g \
        -kernel xen/xen -s -S
 # In separate terminal:
 $ riscv64-buildroot-linux-gnu-gdb
 $ target remote :1234
 $ add-symbol-file <xen_src>/xen/xen-syms 0x80200000
 $ hb *0x80200000
 $ c # it should stop at instruction j 0x80200000 <start>

This suggests to me that Xen is meant to run at VA 0x80200000, whereas ...

Quoted 25 lines
xen/arch/riscv/include/asm/config.h +0 -0
@@ -1,6 +1,9 @@
  #ifndef __RISCV_CONFIG_H__
  #define __RISCV_CONFIG_H__

xen/arch/riscv/include/asm/config.h +3 -0
+#include <xen/const.h>
+#include <xen/page-size.h>
+

 #if defined(CONFIG_RISCV_64)

 # define LONG_BYTEORDER 3
 # define ELFSIZE 64
xen/arch/riscv/include/asm/config.h +0 -0
@@ -28,7 +31,7 @@


 /* Linkage for RISCV */
 #ifdef __ASSEMBLY__

xen/arch/riscv/include/asm/config.h +1 -1
-#define ALIGN .align 2
+#define ALIGN .align 4


 #define ENTRY(name)                                \
   .globl name;                                     \

xen/arch/riscv/include/asm/config.h +0 -0
@@ -36,6 +39,10 @@

   name:
 #endif

Quoted 8 lines
Quoted 1 line

+#define XEN_VIRT_START  _AT(UL, 0x00200000)

... here you specify a much lower address (and to be honest even 0x80200000
looks pretty low to me for 64-bit, and perhaps even for 32-bit). Could you
clarify what the plans here are? Maybe this is merely a temporary thing,
but not called out as such?

It's stale from v1 which had:

#define XEN_VIRT_START  0x80200000

But honestly, I don't think the qemu details in the commit message are
useful.  This series is just about making "make build" work.

The next series (being worked on, but not posted yet) is only a few patches and gets a full Gitlab CI smoke test, at which point the smoke
test shell script is the reference for how to invoke qemu.

Quoted 4 lines

I'm happy to R-by this series and drop that part of the commit message
on commit.

I'm happy with that. Thanks.

Quoted 1 line

~Andrew

author
#5 lore raw

Re: [PATCH v4 1/2] arch/riscv: initial RISC-V support to build/run minimal Xen

Quoted 49 lines · Oleksii
Quoted 48 lines · Andrew Cooper
Quoted 46 lines · Jan Beulich
Quoted 9 lines · Oleksii Kurochko

To run in debug mode should be done the following instructions:

 $ qemu-system-riscv64 -M virt -smp 1 -nographic -m 2g \
        -kernel xen/xen -s -S
 # In separate terminal:
 $ riscv64-buildroot-linux-gnu-gdb
 $ target remote :1234
 $ add-symbol-file <xen_src>/xen/xen-syms 0x80200000
 $ hb *0x80200000
 $ c # it should stop at instruction j 0x80200000 <start>

This suggests to me that Xen is meant to run at VA 0x80200000, whereas ...

Quoted 22 lines
xen/arch/riscv/include/asm/config.h +0 -0
@@ -1,6 +1,9 @@
  #ifndef __RISCV_CONFIG_H__
  #define __RISCV_CONFIG_H__

xen/arch/riscv/include/asm/config.h +3 -0
+#include <xen/const.h>
+#include <xen/page-size.h>
+

 #if defined(CONFIG_RISCV_64)

 # define LONG_BYTEORDER 3
 # define ELFSIZE 64
xen/arch/riscv/include/asm/config.h +0 -0
@@ -28,7 +31,7 @@


 /* Linkage for RISCV */
 #ifdef __ASSEMBLY__

xen/arch/riscv/include/asm/config.h +1 -1
-#define ALIGN .align 2
+#define ALIGN .align 4


 #define ENTRY(name)                                \
   .globl name;                                     \

xen/arch/riscv/include/asm/config.h +0 -0
@@ -36,6 +39,10 @@
Quoted 4 lines

   name:
 #endif

+#define XEN_VIRT_START  _AT(UL, 0x00200000)

... here you specify a much lower address (and to be honest even 0x80200000
looks pretty low to me for 64-bit, and perhaps even for 32-bit). Could you
clarify what the plans here are? Maybe this is merely a temporary thing,
but not called out as such?

Quoted 3 lines
Quoted 3 lines

It's stale from v1 which had:

#define XEN_VIRT_START  0x80200000

Let's switch XEN_VIRT_START to 0x0000000080200000 while we don't have any MMU support as 0x80200000 is an address where OpenSBI will load binary (in our case Xen).

Quoted 12 lines
Quoted 12 lines

But honestly, I don't think the qemu details in the commit message are
useful.  This series is just about making "make build" work.

The next series (being worked on, but not posted yet) is only a few patches and gets a full Gitlab CI smoke test, at which point the smoke
test shell script is the reference for how to invoke qemu.

Quoted 7 lines
Quoted 4 lines

I'm happy to R-by this series and drop that part of the commit message
on commit.

I'm happy with that. Thanks.

Quoted 1 line

~Andrew

~ Oleksii

#6 lore raw

Re: [PATCH v4 1/2] arch/riscv: initial RISC-V support to build/run minimal Xen

Quoted 49 lines · Oleksii
Quoted 48 lines · Oleksii
Quoted 47 lines · Andrew Cooper
Quoted 46 lines · Jan Beulich
Quoted 9 lines · Oleksii Kurochko

To run in debug mode should be done the following instructions:

 $ qemu-system-riscv64 -M virt -smp 1 -nographic -m 2g \
        -kernel xen/xen -s -S
 # In separate terminal:
 $ riscv64-buildroot-linux-gnu-gdb
 $ target remote :1234
 $ add-symbol-file <xen_src>/xen/xen-syms 0x80200000
 $ hb *0x80200000
 $ c # it should stop at instruction j 0x80200000 <start>

This suggests to me that Xen is meant to run at VA 0x80200000, whereas ...

Quoted 22 lines
xen/arch/riscv/include/asm/config.h +0 -0
@@ -1,6 +1,9 @@
  #ifndef __RISCV_CONFIG_H__
  #define __RISCV_CONFIG_H__

xen/arch/riscv/include/asm/config.h +3 -0
+#include <xen/const.h>
+#include <xen/page-size.h>
+

 #if defined(CONFIG_RISCV_64)

 # define LONG_BYTEORDER 3
 # define ELFSIZE 64
xen/arch/riscv/include/asm/config.h +0 -0
@@ -28,7 +31,7 @@


 /* Linkage for RISCV */
 #ifdef __ASSEMBLY__

xen/arch/riscv/include/asm/config.h +1 -1
-#define ALIGN .align 2
+#define ALIGN .align 4


 #define ENTRY(name)                                \
   .globl name;                                     \

xen/arch/riscv/include/asm/config.h +0 -0
@@ -36,6 +39,10 @@
Quoted 4 lines

   name:
 #endif

+#define XEN_VIRT_START  _AT(UL, 0x00200000)

... here you specify a much lower address (and to be honest even 0x80200000
looks pretty low to me for 64-bit, and perhaps even for 32-bit). Could you
clarify what the plans here are? Maybe this is merely a temporary thing,
but not called out as such?

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

It's stale from v1 which had:

#define XEN_VIRT_START  0x80200000

Let's switch XEN_VIRT_START to 0x0000000080200000 while we don't have any MMU support as 0x80200000 is an address where OpenSBI will load binary (in our case Xen).

Ok.  I've fixed that up and pushed with a tweaked commit message.

~Andrew