Initial support for Power
v4 cover Initial support for Power
Hello all,
This patch series adds support for building a minimal image for Power ISA 2.07B+ (POWER8+) systems. In addition to a patch adding support to the build system and a simple infinite loop at the entrypoint, patches to add ppc64le support to the CI as well as a MAINTAINERS update are included.
Since Xen previously had support for a much older version of the ISA in version 3.2.3, we were able to carry over some headers and support routines from that version. Unlike that initial port though, this effort focuses solely on POWER8+ CPUs that are capable of running in Little Endian mode.
With an appropriate powerpc64le-linux-gnu cross-toolchain, the minimal image can be built with:
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen build
The resulting binary can then be booted in a standard QEMU/pseries VM:
$ qemu-system-ppc64 -M pseries-5.2 -m 256M -kernel xen/xen \ -vga none -serial mon:stdio -nographic
Thanks,
Shawn
--
Changes from v4:
- Change '$(@D)/$(@F)' to '$@' in ppc/Makefile
- Add -m64, -mlittle-endian to CFLAGS to allow using ppc{32,64be} toolchains
- Change IBM-specific '$' to '.' in head.S
- Drop unnecessary ALIGN() in linker script in .bss
Changes from v3:
- Fix formatting of MAINTAINERS patch
Changes from v2:
- Add ppc64le cross-build container patch
- Add ppc64le cross build CI job patch
- Drop serial output patch (will be in future patch series)
- Drop setup.c and unneeded headers from minimal build patch
- Fixed ordering of MAINTAINERS patch + add F: line
- Fix config/ppc64.mk option names
- Clarify Kconfig Baseline ISA option help strings
Shawn Anastasio (4):
automation: Add container for ppc64le builds
xen: Add files needed for minimal ppc64le build
automation: Add ppc64le cross-build jobs
maintainers: Add ppc64 maintainer
MAINTAINERS | 4 + .../build/debian/bullseye-ppc64le.dockerfile | 28 +++ automation/gitlab-ci/build.yaml | 60 ++++++ automation/scripts/containerize | 1 + config/ppc64.mk | 5 + xen/Makefile | 5 +- xen/arch/ppc/Kconfig | 42 +++++ xen/arch/ppc/Kconfig.debug | 0 xen/arch/ppc/Makefile | 16 ++ xen/arch/ppc/Rules.mk | 0 xen/arch/ppc/arch.mk | 12 ++ xen/arch/ppc/configs/openpower_defconfig | 13 ++ xen/arch/ppc/include/asm/config.h | 63 +++++++ xen/arch/ppc/include/asm/page-bits.h | 7 + xen/arch/ppc/ppc64/Makefile | 1 + xen/arch/ppc/ppc64/asm-offsets.c | 0 xen/arch/ppc/ppc64/head.S | 27 +++ xen/arch/ppc/xen.lds.S | 172 ++++++++++++++++++ 18 files changed, 454 insertions(+), 2 deletions(-) create mode 100644 automation/build/debian/bullseye-ppc64le.dockerfile create mode 100644 config/ppc64.mk create mode 100644 xen/arch/ppc/Kconfig create mode 100644 xen/arch/ppc/Kconfig.debug create mode 100644 xen/arch/ppc/Makefile create mode 100644 xen/arch/ppc/Rules.mk create mode 100644 xen/arch/ppc/arch.mk create mode 100644 xen/arch/ppc/configs/openpower_defconfig create mode 100644 xen/arch/ppc/include/asm/config.h create mode 100644 xen/arch/ppc/include/asm/page-bits.h create mode 100644 xen/arch/ppc/ppc64/Makefile create mode 100644 xen/arch/ppc/ppc64/asm-offsets.c create mode 100644 xen/arch/ppc/ppc64/head.S create mode 100644 xen/arch/ppc/xen.lds.S
--
2.30.2
PATCH v4 1/4 automation: Add container for ppc64le builds
Add a container for cross-compiling xen for ppc64le.
--- .../build/debian/bullseye-ppc64le.dockerfile | 28 +++++++++++++++++++ automation/scripts/containerize | 1 + 2 files changed, 29 insertions(+) create mode 100644 automation/build/debian/bullseye-ppc64le.dockerfile
automation/build/debian/bullseye-ppc64le.dockerfile new file +28 -0
@@ -0,0 +1,28 @@ +FROM debian:bullseye-slim +LABEL maintainer.name="The Xen Project" \ + maintainer.email="[email protected]" + +ENV DEBIAN_FRONTEND=noninteractive +ENV USER root + +# Add compiler path +ENV CROSS_COMPILE powerpc64le-linux-gnu- + +RUN mkdir /build +WORKDIR /build + +# build depends +RUN apt-get update && \ + apt-get --quiet --yes --no-install-recommends install \ + bison \ + build-essential \ + checkpolicy \ + flex \ + gawk \ + gcc-powerpc64le-linux-gnu \ + make \ + python3-minimal \ + && \ + apt-get autoremove -y && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
automation/scripts/containerize +1 -0
@@ -33,6 +33,7 @@ case "_${CONTAINER}" in _focal) CONTAINER="${BASE}/ubuntu:focal" ;; _jessie) CONTAINER="${BASE}/debian:jessie" ;; _jessie-i386) CONTAINER="${BASE}/debian:jessie-i386" ;; + _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;; _stretch|_) CONTAINER="${BASE}/debian:stretch" ;; _stretch-i386) CONTAINER="${BASE}/debian:stretch-i386" ;; _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
Signature · 1 line
2.30.2
Re: [PATCH v4 1/4] automation: Add container for ppc64le builds
Quoted 4 lines · Shawn Anastasio
Add a container for cross-compiling xen for ppc64le.
Signed-off-by Shawn Anastasio <[email protected]>Acked-by Andrew Cooper <[email protected]>
Quoted 55 lines
--- .../build/debian/bullseye-ppc64le.dockerfile | 28 +++++++++++++++++++ automation/scripts/containerize | 1 + 2 files changed, 29 insertions(+) create mode 100644 automation/build/debian/bullseye-ppc64le.dockerfileautomation/build/debian/bullseye-ppc64le.dockerfile new file +28
-0@@ -0,0 +1,28 @@ +FROM debian:bullseye-slim +LABEL maintainer.name="The Xen Project" \ + maintainer.email="[email protected]" + +ENV DEBIAN_FRONTEND=noninteractive +ENV USER root + +# Add compiler path +ENV CROSS_COMPILE powerpc64le-linux-gnu- + +RUN mkdir /build +WORKDIR /build + +# build depends +RUN apt-get update && \ + apt-get --quiet --yes --no-install-recommends install \ + bison \ + build-essential \ + checkpolicy \ + flex \ + gawk \ + gcc-powerpc64le-linux-gnu \ + make \ + python3-minimal \ + && \ + apt-get autoremove -y && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*automation/scripts/containerize +1
-0@@ -33,6 +33,7 @@ case "_${CONTAINER}" in _focal) CONTAINER="${BASE}/ubuntu:focal" ;; _jessie) CONTAINER="${BASE}/debian:jessie" ;; _jessie-i386) CONTAINER="${BASE}/debian:jessie-i386" ;; + _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;; _stretch|_) CONTAINER="${BASE}/debian:stretch" ;; _stretch-i386) CONTAINER="${BASE}/debian:stretch-i386" ;; _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;Signature · 1 line
2.30.2
Re: [PATCH v4 1/4] automation: Add container for ppc64le builds
In reply to Shawn AnastasioQuoted 4 lines · Shawn Anastasio
Add a container for cross-compiling xen for ppc64le.
Signed-off-by Shawn Anastasio <[email protected]>Acked-by Andrew Cooper <[email protected]>
Just to say that I already committed this patch, when rebuilding the container. We likely raced given the time you posted the series, but you'll want to drop it as part of rebasing onto the staging branch (if another series is needed).
~Andrew
Re: [PATCH v4 1/4] automation: Add container for ppc64le builds
Quoted 9 lines · Andrew Cooper
Quoted 4 lines · Shawn Anastasio
Add a container for cross-compiling xen for ppc64le.
Signed-off-by Shawn Anastasio <[email protected]>Acked-by Andrew Cooper <[email protected]>Just to say that I already committed this patch, when rebuilding the container. We likely raced given the time you posted the series, but you'll want to drop it as part of rebasing onto the staging branch (if another series is needed).
Apologies. If a v5 is needed I'll definitely exclude this patch then.
Quoted 1 line
~Andrew
Thanks,
Shawn
PATCH v4 2/4 xen: Add files needed for minimal ppc64le build
In reply to Shawn AnastasioAdd the build system changes required to build for ppc64le (POWER8+). As of now the resulting image simply boots to an infinite loop.
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen build
This port targets POWER8+ CPUs running in Little Endian mode specifically, and does not boot on older machines. Additionally, this initial skeleton only implements the PaPR/pseries boot protocol which allows it to be booted in a standard QEMU virtual machine:
$ qemu-system-ppc64 -M pseries-5.2 -m 256M -kernel xen/xen
--- config/ppc64.mk | 5 + xen/Makefile | 5 +- xen/arch/ppc/Kconfig | 42 ++++++ xen/arch/ppc/Kconfig.debug | 0 xen/arch/ppc/Makefile | 16 +++ xen/arch/ppc/Rules.mk | 0 xen/arch/ppc/arch.mk | 12 ++ xen/arch/ppc/configs/openpower_defconfig | 13 ++ xen/arch/ppc/include/asm/config.h | 63 +++++++++ xen/arch/ppc/include/asm/page-bits.h | 7 + xen/arch/ppc/ppc64/Makefile | 1 + xen/arch/ppc/ppc64/asm-offsets.c | 0 xen/arch/ppc/ppc64/head.S | 27 ++++ xen/arch/ppc/xen.lds.S | 172 +++++++++++++++++++++++ 14 files changed, 361 insertions(+), 2 deletions(-) create mode 100644 config/ppc64.mk create mode 100644 xen/arch/ppc/Kconfig create mode 100644 xen/arch/ppc/Kconfig.debug create mode 100644 xen/arch/ppc/Makefile create mode 100644 xen/arch/ppc/Rules.mk create mode 100644 xen/arch/ppc/arch.mk create mode 100644 xen/arch/ppc/configs/openpower_defconfig create mode 100644 xen/arch/ppc/include/asm/config.h create mode 100644 xen/arch/ppc/include/asm/page-bits.h create mode 100644 xen/arch/ppc/ppc64/Makefile create mode 100644 xen/arch/ppc/ppc64/asm-offsets.c create mode 100644 xen/arch/ppc/ppc64/head.S create mode 100644 xen/arch/ppc/xen.lds.S
config/ppc64.mk new file +5 -0
@@ -0,0 +1,5 @@ +CONFIG_PPC := y +CONFIG_PPC64 := y +CONFIG_PPC_$(XEN_OS) := y + +CONFIG_XEN_INSTALL_SUFFIX :=
xen/Makefile +3 -2
@@ -38,7 +38,7 @@ EFI_MOUNTPOINT ?= $(BOOT_DIR)/efi ARCH=$(XEN_TARGET_ARCH) SRCARCH=$(shell echo $(ARCH) | \ sed -e 's/x86.*/x86/' -e s'/arm\(32\|64\)/arm/g' \ - -e s'/riscv.*/riscv/g') + -e s'/riscv.*/riscv/g' -e s'/ppc.*/ppc/g') export ARCH SRCARCH # Allow someone to change their config file
@@ -244,7 +244,7 @@ include $(XEN_ROOT)/Config.mk export TARGET_SUBARCH := $(XEN_TARGET_ARCH) export TARGET_ARCH := $(shell echo $(XEN_TARGET_ARCH) | \ sed -e 's/x86.*/x86/' -e s'/arm\(32\|64\)/arm/g' \ - -e s'/riscv.*/riscv/g') + -e s'/riscv.*/riscv/g' -e s'/ppc.*/ppc/g') export CONFIG_SHELL := $(SHELL) export CC CXX LD NM OBJCOPY OBJDUMP ADDR2LINE
@@ -563,6 +563,7 @@ _clean: $(Q)$(MAKE) $(clean)=xsm $(Q)$(MAKE) $(clean)=crypto $(Q)$(MAKE) $(clean)=arch/arm + $(Q)$(MAKE) $(clean)=arch/ppc $(Q)$(MAKE) $(clean)=arch/riscv $(Q)$(MAKE) $(clean)=arch/x86 $(Q)$(MAKE) $(clean)=test
xen/arch/ppc/Kconfig new file +42 -0
42 lines not on this page · raw message
xen/arch/ppc/Kconfig.debug new file +0 -0
xen/arch/ppc/Makefile new file +16 -0
@@ -0,0 +1,16 @@ +obj-$(CONFIG_PPC64) += ppc64/ + +$(TARGET): $(TARGET)-syms + cp -f $< $@ + +$(TARGET)-syms: $(objtree)/prelink.o $(obj)/xen.lds + $(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) -o $@ + $(NM) -pa --format=sysv $@ \ + | $(objtree)/tools/symbols --all-symbols --xensyms --sysv --sort \ + > [email protected] + +$(obj)/xen.lds: $(src)/xen.lds.S FORCE + $(call if_changed_dep,cpp_lds_S) + +.PHONY: include +include:
xen/arch/ppc/Rules.mk new file +0 -0
xen/arch/ppc/arch.mk new file +12 -0
@@ -0,0 +1,12 @@ +######################################## +# Power-specific definitions + +ppc-march-$(CONFIG_POWER_ISA_2_07B) := power8 +ppc-march-$(CONFIG_POWER_ISA_3_00) := power9 + +CFLAGS += -m64 -mlittle-endian -mcpu=$(ppc-march-y) +CFLAGS += -mstrict-align -mcmodel=large -mabi=elfv2 -mno-altivec -mno-vsx + +# TODO: Drop override when more of the build is working +override ALL_OBJS-y = arch/$(TARGET_ARCH)/built_in.o +override ALL_LIBS-y =
xen/arch/ppc/configs/openpower_defconfig new file +13 -0
@@ -0,0 +1,13 @@ +# CONFIG_SCHED_CREDIT is not set +# CONFIG_SCHED_RTDS is not set +# CONFIG_SCHED_NULL is not set +# CONFIG_SCHED_ARINC653 is not set +# CONFIG_TRACEBUFFER is not set +# CONFIG_HYPFS is not set +# CONFIG_GRANT_TABLE is not set +# CONFIG_SPECULATIVE_HARDEN_ARRAY is not set + +CONFIG_PPC64=y +CONFIG_DEBUG=y +CONFIG_DEBUG_INFO=y +CONFIG_EXPERT=y
xen/arch/ppc/include/asm/config.h new file +63 -0
63 lines not on this page · raw message
xen/arch/ppc/include/asm/page-bits.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef __PPC_PAGE_BITS_H__ +#define __PPC_PAGE_BITS_H__ + +#define PAGE_SHIFT 16/* 64 KiB Pages */ +#define PADDR_BITS 48 + +#endif /* __PPC_PAGE_BITS_H__ */
xen/arch/ppc/ppc64/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-y += head.o
xen/arch/ppc/ppc64/asm-offsets.c new file +0 -0
xen/arch/ppc/ppc64/head.S new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */ + tdi 0, 0, 0x48 /* Reverse endian of b . + 8 */ + b . + 44 /* Skip trampoline if endian is good */ + .long 0xa600607d /* mfmsr r11 */ + .long 0x01006b69 /* xori r11,r11,1 */ + .long 0x00004039 /* li r10,0 */ + .long 0x6401417d /* mtmsrd r10,1 */ + .long 0x05009f42 /* bcl 20,31,$+4 */ + .long 0xa602487d /* mflr r10 */ + .long 0x14004a39 /* addi r10,r10,20 */ + .long 0xa6035a7d /* mtsrr0 r10 */ + .long 0xa6037b7d /* mtsrr1 r11 */ + .long 0x2400004c /* rfid */ + + /* Now that the endianness is confirmed, continue */ +1: b 1b
xen/arch/ppc/xen.lds.S new file +172 -0
172 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 5 lines · Shawn Anastasio
Add the build system changes required to build for ppc64le (POWER8+). As of now the resulting image simply boots to an infinite loop.
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen build
I think the first of these isn't needed, given the config ARCH_DEFCONFIG default. I'd suggest dropping it.
On the second, what is the SUBSYSTEMS=xen? It's not needed given the stripped down build system, but I don't see why we'd ever be compiling Xen with some kind of subsystem configuration for something else.
config/ppc64.mk line 3
Quoted 9 lines
config/ppc64.mk new file +3
-0@@ -0,0 +1,5 @@ +CONFIG_PPC := y +CONFIG_PPC64 := y +CONFIG_PPC_$(XEN_OS) := y
I know you're copying the existing architectures, but I'm pretty certain these $(XEN_OS) expressions are pretty bogus. The userspace stuff in tools/ may need to know the host OS it's being built for, but Xen really doesn't.
I'm pretty sure it will compile with this dropped, so please do. I'll see about patching it out of the other architectures.
Quoted 30 lines
xen/arch/ppc/Kconfig new file +24
-0@@ -0,0 +1,42 @@ +config PPC + def_bool y + +config PPC64 + def_bool y + select 64BIT + +config ARCH_DEFCONFIG + string + default "arch/ppc/configs/openpower_defconfig" + +menu "Architecture Features" + +source "arch/Kconfig" + +endmenu + +menu "ISA Selection" + +choice + prompt "Base ISA" + default POWER_ISA_2_07B if PPC64 + help + This selects the base ISA version that Xen will target.
xen/arch/ppc/Kconfig line 34
Quoted 10 lines
xen/arch/ppc/Kconfig new file +10
-0+ +config POWER_ISA_2_07B + bool "Power ISA 2.07B" + help + Target version 2.07B of the Power ISA (POWER8) + +config POWER_ISA_3_00 + bool "Power ISA 3.00" + help + Target version 3.00 of the Power ISA (POWER9)
For both of these, it will be helpful for anyone who isn't as PPC-knowledgeable if the POWER8/9 was in the title too, seeing as they're the most common name.
But as I'm a noob here too, how different are Power8 and 9? Given they share a head.S, they're presumably not too disjoint in terms of ISA.
While being able to target a specific CPU is something we're trying to retrofit to Xen, by default we do expect it to run on as broad a set of systems as possible.
If that's not feasible, then fine, but if it is, it ought to be the default. Which might be as simple as saying "or later" somewhere in this text, or might be a giant can of worms that I shouldn't open...
xen/arch/ppc/include/asm/page-bits.h line 5
Quoted 11 lines
xen/arch/ppc/include/asm/page-bits.h new file +5
-0@@ -0,0 +1,7 @@ +#ifndef __PPC_PAGE_BITS_H__ +#define __PPC_PAGE_BITS_H__ + +#define PAGE_SHIFT 16/* 64 KiB Pages */ +#define PADDR_BITS 48
Is 64k the minimum granularity? Or is 4k an option?
I ask because Xen has some very short sighted ABIs which we're still working on removing. There are still quite a few expectations of PAGE_SHIFT being 12.
To be clear, we're looking to fix all of these ABIs, but I suspect it will be an easier lift to begin with at 4k. (Or perhaps the right thing is to double down and just get them fixed.)
Quoted 35 lines
xen/arch/ppc/include/asm/page-bits.h new file +2
-0+ +#endif /* __PPC_PAGE_BITS_H__ */ xen/arch/ppc/ppc64/Makefile new file +1
-0@@ -0,0 +1 @@ +obj-y += head.oxen/arch/ppc/ppc64/asm-offsets.c new file +0
-0xen/arch/ppc/ppc64/head.S new file +17
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */ + tdi 0, 0, 0x48 /* Reverse endian of b . + 8 */ + b . + 44 /* Skip trampoline if endian is good */ + .long 0xa600607d /* mfmsr r11 */ + .long 0x01006b69 /* xori r11,r11,1 */ + .long 0x00004039 /* li r10,0 */
xen/arch/ppc/ppc64/head.S line 27
Quoted 10 lines
xen/arch/ppc/ppc64/head.S new file +10
-0+ .long 0x6401417d /* mtmsrd r10,1 */ + .long 0x05009f42 /* bcl 20,31,$+4 */ + .long 0xa602487d /* mflr r10 */ + .long 0x14004a39 /* addi r10,r10,20 */ + .long 0xa6035a7d /* mtsrr0 r10 */ + .long 0xa6037b7d /* mtsrr1 r11 */ + .long 0x2400004c /* rfid */ + + /* Now that the endianness is confirmed, continue */ +1: b 1b
.size start, . - start
.type start, %function
Lets get the ELF metadata right from the start.
xen/arch/ppc/xen.lds.S line 172
Quoted 13 lines
xen/arch/ppc/xen.lds.S new file +6
-0@@ -0,0 +1,172 @@ + DISCARD_SECTIONS + + STABS_DEBUG_SECTIONS + + ELF_DETAILS_SECTIONS +}
In the other architectures, we now assert that sections such as .got are empty, because we've had enough bugs in the past.
I'd recommend doing the same from the outset for all the dynamic relocation sections, unless you're expecting to have to support them?
~Andrew
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Hi,
Quoted 25 lines · Andrew Cooper
Quoted 25 lines
xen/arch/ppc/ppc64/head.S new file +22
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */ + tdi 0, 0, 0x48 /* Reverse endian of b . + 8 */ + b . + 44 /* Skip trampoline if endian is good */ + .long 0xa600607d /* mfmsr r11 */ + .long 0x01006b69 /* xori r11,r11,1 */ + .long 0x00004039 /* li r10,0 */ + .long 0x6401417d /* mtmsrd r10,1 */ + .long 0x05009f42 /* bcl 20,31,$+4 */ + .long 0xa602487d /* mflr r10 */ + .long 0x14004a39 /* addi r10,r10,20 */ + .long 0xa6035a7d /* mtsrr0 r10 */
Quoted 8 lines
xen/arch/ppc/ppc64/head.S line 27
Quoted 5 lines
xen/arch/ppc/ppc64/head.S new file +5
-0+ .long 0xa6037b7d /* mtsrr1 r11 */ + .long 0x2400004c /* rfid */ + + /* Now that the endianness is confirmed, continue */ +1: b 1b.size start, . - start
.type start, %function
Shouldn't we introduce ENDPROC()/END() to avoid open-coding these two lines everywhere?
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 35 lines · Julien Grall
Quoted 25 lines · Andrew Cooper
Quoted 25 lines
xen/arch/ppc/ppc64/head.S new file +22
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */ + tdi 0, 0, 0x48 /* Reverse endian of b . + 8 */ + b . + 44 /* Skip trampoline if endian is good */ + .long 0xa600607d /* mfmsr r11 */ + .long 0x01006b69 /* xori r11,r11,1 */ + .long 0x00004039 /* li r10,0 */ + .long 0x6401417d /* mtmsrd r10,1 */ + .long 0x05009f42 /* bcl 20,31,$+4 */ + .long 0xa602487d /* mflr r10 */ + .long 0x14004a39 /* addi r10,r10,20 */ + .long 0xa6035a7d /* mtsrr0 r10 */Quoted 8 lines
xen/arch/ppc/ppc64/head.S line 27
Quoted 5 lines
xen/arch/ppc/ppc64/head.S new file +5
-0+ .long 0xa6037b7d /* mtsrr1 r11 */ + .long 0x2400004c /* rfid */ + + /* Now that the endianness is confirmed, continue */ +1: b 1b.size start, . - start
.type start, %function
Quoted 2 lines
Shouldn't we introduce ENDPROC()/END() to avoid open-coding these two lines everywhere?
A proposal for this is pending, see "x86: annotate entry points with type and size". That proposal extends to asking whether to use this scheme uniformly, not just for x86.
Jan
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
In reply to Andrew CooperQuoted 9 lines · Andrew Cooper
Quoted 5 lines · Shawn Anastasio
Add the build system changes required to build for ppc64le (POWER8+). As of now the resulting image simply boots to an infinite loop.
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen buildI think the first of these isn't needed, given the config
ARCH_DEFCONFIGdefault. I'd suggest dropping it.
It seems like the build system expects an $(ARCH)_defconfig present if you don't manually specify a defconfig target. I see riscv64 has a tiny64_defconfig and a riscv64_defconfig that are idential, probably for this same reason. Would you like me to take the same approach of duplicating openpower_defconfig to ppc64_defconfig?
Quoted 3 lines
On the second, what is the SUBSYSTEMS=xen? It's not needed given the stripped down build system, but I don't see why we'd ever be compiling Xen with some kind of subsystem configuration for something else.
This was a remnant of the old head.o-only TARGET build command that I got from the initial riscv commit. You're correct that it's unnecessary now and I'll drop it from the commit message.
Quoted 15 lines
config/ppc64.mk line 3
Quoted 9 lines
config/ppc64.mk new file +3
-0@@ -0,0 +1,5 @@ +CONFIG_PPC := y +CONFIG_PPC64 := y +CONFIG_PPC_$(XEN_OS) := yI know you're copying the existing architectures, but I'm pretty certain these $(XEN_OS) expressions are pretty bogus. The userspace stuff in
tools/may need to know the host OS it's being built for, but Xen really doesn't.
Quoted 2 lines
I'm pretty sure it will compile with this dropped, so please do. I'll see about patching it out of the other architectures.
Sure, I'll drop this in v5.
Quoted 45 lines
Quoted 30 lines
xen/arch/ppc/Kconfig new file +24
-0@@ -0,0 +1,42 @@ +config PPC + def_bool y + +config PPC64 + def_bool y + select 64BIT + +config ARCH_DEFCONFIG + string + default "arch/ppc/configs/openpower_defconfig" + +menu "Architecture Features" + +source "arch/Kconfig" + +endmenu + +menu "ISA Selection" + +choice + prompt "Base ISA" + default POWER_ISA_2_07B if PPC64 + help + This selects the base ISA version that Xen will target.xen/arch/ppc/Kconfig line 34
Quoted 10 lines
xen/arch/ppc/Kconfig new file +10
-0+ +config POWER_ISA_2_07B + bool "Power ISA 2.07B" + help + Target version 2.07B of the Power ISA (POWER8) + +config POWER_ISA_3_00 + bool "Power ISA 3.00" + help + Target version 3.00 of the Power ISA (POWER9)For both of these, it will be helpful for anyone who isn't as PPC-knowledgeable if the POWER8/9 was in the title too, seeing as they're the most common name.
Quoted 6 lines
But as I'm a noob here too, how different are Power8 and 9? Given they share a
head.S, they're presumably not too disjoint in terms of ISA.While being able to target a specific CPU is something we're trying to retrofit to Xen, by default we do expect it to run on as broad a set of systems as possible.
They're not that different, and a kernel built for POWER8 should just work on POWER9. The intent was to specify a baseline feature set that guards whether, i.e., POWER9/ISA3.0-only features should be built.
Quoted 3 lines
If that's not feasible, then fine, but if it is, it ought to be the default. Which might be as simple as saying "or later" somewhere in this text, or might be a giant can of worms that I shouldn't open...
Originally the help text for the two ISA config options ended in a "+" but that was deemed ambiguous. Would adding "or later" to the help text for the two options clarify it sufficiently?
Quoted 5 lines
Quoted 5 lines
xen/arch/ppc/include/asm/page-bits.h new file +0
-0
Quoted 8 lines
xen/arch/ppc/include/asm/page-bits.h line 5
Quoted 6 lines
xen/arch/ppc/include/asm/page-bits.h new file +5
-0@@ -0,0 +1,7 @@ +#ifndef __PPC_PAGE_BITS_H__ +#define __PPC_PAGE_BITS_H__ + +#define PAGE_SHIFT 16/* 64 KiB Pages */ +#define PADDR_BITS 48Is 64k the minimum granularity? Or is 4k an option?
Both 4K and 64K are supported by the hardware.
Quoted 7 lines
I ask because Xen has some very short sighted ABIs which we're still working on removing. There are still quite a few expectations of
PAGE_SHIFTbeing 12.To be clear, we're looking to fix all of these ABIs, but I suspect it will be an easier lift to begin with at 4k. (Or perhaps the right thing is to double down and just get them fixed.)
Interesting. Given this I'm inclined to go with 4k just to reduce pain points during initial bring up, though supporting 64k would definitely be desirable going forward.
Quoted 49 lines
Quoted 35 lines
xen/arch/ppc/include/asm/page-bits.h new file +2
-0+ +#endif /* __PPC_PAGE_BITS_H__ */ xen/arch/ppc/ppc64/Makefile new file +1
-0@@ -0,0 +1 @@ +obj-y += head.oxen/arch/ppc/ppc64/asm-offsets.c new file +0
-0xen/arch/ppc/ppc64/head.S new file +17
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */ + tdi 0, 0, 0x48 /* Reverse endian of b . + 8 */ + b . + 44 /* Skip trampoline if endian is good */ + .long 0xa600607d /* mfmsr r11 */ + .long 0x01006b69 /* xori r11,r11,1 */ + .long 0x00004039 /* li r10,0 */xen/arch/ppc/ppc64/head.S line 27
Quoted 10 lines
xen/arch/ppc/ppc64/head.S new file +10
-0+ .long 0x6401417d /* mtmsrd r10,1 */ + .long 0x05009f42 /* bcl 20,31,$+4 */ + .long 0xa602487d /* mflr r10 */ + .long 0x14004a39 /* addi r10,r10,20 */ + .long 0xa6035a7d /* mtsrr0 r10 */ + .long 0xa6037b7d /* mtsrr1 r11 */ + .long 0x2400004c /* rfid */ + + /* Now that the endianness is confirmed, continue */ +1: b 1b.size start, . - start
.type start, %function
Quoted 1 line
Lets get the ELF metadata right from the start.
Good point. Following the example in the Power ELFv2 ABI specification [1] for function type annotation, I'll place
.type start, @function
in the ENTRY macro. It's not clear what the difference between %function and @function are in this context (my toolchain seems to accept both and produce the same ELF metadata), but the latter is more idiomatic in Power assembly. The same goes for its placement before the entrypoint vs. after the last instruction.
As for the size annotation, I'll follow Julien's suggestion and introduce an END macro.
Quoted 17 lines
xen/arch/ppc/xen.lds.S line 172
Quoted 13 lines
xen/arch/ppc/xen.lds.S new file +6
-0@@ -0,0 +1,172 @@ + DISCARD_SECTIONS + + STABS_DEBUG_SECTIONS + + ELF_DETAILS_SECTIONS +}In the other architectures, we now assert that sections such as .got are empty, because we've had enough bugs in the past.
Quoted 2 lines
I'd recommend doing the same from the outset for all the dynamic relocation sections, unless you're expecting to have to support them?
No plans on supporting dynamic relocation (for now), so I can go ahead and add these assertions.
Quoted 1 line
~Andrew
Thanks,
Shawn
[1] Page 77 https://wiki.raptorcs.com/w/images/7/70/Leabi-20170510.pdf
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 8 lines · Shawn Anastasio
Quoted 7 lines · Andrew Cooper
Quoted 5 lines · Shawn Anastasio
Add the build system changes required to build for ppc64le (POWER8+). As of now the resulting image simply boots to an infinite loop.
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen build
Quoted 8 lines
Quoted 2 lines
I think the first of these isn't needed, given the config
ARCH_DEFCONFIGdefault. I'd suggest dropping it.It seems like the build system expects an
$(ARCH)_defconfigpresent if you don't manually specify a defconfig target. I see riscv64 has atiny64_defconfigand ariscv64_defconfigthat are idential, probably for this same reason. Would you like me to take the same approach of duplicatingopenpower_defconfigtoppc64_defconfig?
It's a symlink for RISC-V iirc, so wants to be that same way for PPC then (for the time being, again like there).
Quoted 39 lines
Quoted 34 lines
Quoted 20 lines
xen/arch/ppc/ppc64/head.S new file +17
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */ + tdi 0, 0, 0x48 /* Reverse endian of b . + 8 */ + b . + 44 /* Skip trampoline if endian is good */ + .long 0xa600607d /* mfmsr r11 */ + .long 0x01006b69 /* xori r11,r11,1 */ + .long 0x00004039 /* li r10,0 */xen/arch/ppc/ppc64/head.S line 27
Quoted 10 lines
xen/arch/ppc/ppc64/head.S new file +10
-0+ .long 0x6401417d /* mtmsrd r10,1 */ + .long 0x05009f42 /* bcl 20,31,$+4 */ + .long 0xa602487d /* mflr r10 */ + .long 0x14004a39 /* addi r10,r10,20 */ + .long 0xa6035a7d /* mtsrr0 r10 */ + .long 0xa6037b7d /* mtsrr1 r11 */ + .long 0x2400004c /* rfid */ + + /* Now that the endianness is confirmed, continue */ +1: b 1b.size start, . - start
.type start, %functionQuoted 1 line
Lets get the ELF metadata right from the start.
Good point. Following the example in the Power ELFv2 ABI specification [1] for function type annotation, I'll place
Quoted 7 lines
.type start, @function
in the ENTRY macro. It's not clear what the difference between %function and @function are in this context (my toolchain seems to accept both and produce the same ELF metadata), but the latter is more idiomatic in Power assembly. The same goes for its placement before the entrypoint vs. after the last instruction.
% and @ are entirely the same here, except for targets like arm-*, where @ starts a comment.
Quoted 14 lines
Quoted 14 lines
xen/arch/ppc/xen.lds.S line 172
Quoted 10 lines
xen/arch/ppc/xen.lds.S new file +6
-0@@ -0,0 +1,172 @@ + DISCARD_SECTIONS + + STABS_DEBUG_SECTIONS + + ELF_DETAILS_SECTIONS +}In the other architectures, we now assert that sections such as .got are empty, because we've had enough bugs in the past.
Quoted 5 lines
Quoted 2 lines
I'd recommend doing the same from the outset for all the dynamic relocation sections, unless you're expecting to have to support them?
No plans on supporting dynamic relocation (for now), so I can go ahead and add these assertions.
How would you ever use dynamic relocations? You have no loader to process them for you. (Yes, there can of course be relocation free head.S code which deals with relocations, but doing such in assembly is likely not the best idea. Yet as soon as you enter C code you're at risk of hitting a place requiring such a relocation, perhaps simply because of a "careless" function call on some infrequently used code path.)
Jan
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 8 lines · Jan Beulich
Quoted 5 lines
It seems like the build system expects an
$(ARCH)_defconfigpresent if you don't manually specify a defconfig target. I see riscv64 has atiny64_defconfigand ariscv64_defconfigthat are idential, probably for this same reason. Would you like me to take the same approach of duplicatingopenpower_defconfigtoppc64_defconfig?It's a symlink for RISC-V iirc, so wants to be that same way for PPC then (for the time being, again like there).
Ah, you're right, riscv uses a symlink. I'll use that approach here as well then.
Quoted 5 lines
Quoted 5 lines
Good point. Following the example in the Power ELFv2 ABI specification [1] for function type annotation, I'll place
.type start, @function
Quoted 8 lines
Quoted 5 lines
in the ENTRY macro. It's not clear what the difference between %function and @function are in this context (my toolchain seems to accept both and produce the same ELF metadata), but the latter is more idiomatic in Power assembly. The same goes for its placement before the entrypoint vs. after the last instruction.
% and @ are entirely the same here, except for targets like arm-*, where @ starts a comment.
Okay, thanks for the confirmation. Is there a preference for which one is used? I'd lean towards @ just for the sake of remaining idiomatic, but since they're equivalent it doesn't really matter from my perspective.
Quoted 10 lines
Quoted 2 lines
No plans on supporting dynamic relocation (for now), so I can go ahead and add these assertions.
How would you ever use dynamic relocations? You have no loader to process them for you. (Yes, there can of course be relocation free
head.Scode which deals with relocations, but doing such in assembly is likely not the best idea. Yet as soon as you enter C code you're at risk of hitting a place requiring such a relocation, perhaps simply because of a "careless" function call on some infrequently used code path.)
Right, those concerns are definitely something we'd have to deal with if we ever wanted to add support for relocations (and a large part of the reason why I said I don't have plans to support it right now).
As an aside, I'll note that Linux uses the approach of doing the relocation fixups entirely in assembly before the C entrypoint, but that's obviously not a super desirable approach from a correctness or maintainability standpoint.
Quoted 1 line
Jan
Thanks,
Shawn
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 12 lines · Shawn Anastasio
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Good point. Following the example in the Power ELFv2 ABI specification [1] for function type annotation, I'll place
.type start, @function
in the ENTRY macro. It's not clear what the difference between %function and @function are in this context (my toolchain seems to accept both and produce the same ELF metadata), but the latter is more idiomatic in Power assembly. The same goes for its placement before the entrypoint vs. after the last instruction.
Quoted 7 lines
Quoted 2 lines
% and @ are entirely the same here, except for targets like arm-*, where @ starts a comment.
Okay, thanks for the confirmation. Is there a preference for which one is used? I'd lean towards @ just for the sake of remaining idiomatic, but since they're equivalent it doesn't really matter from my perspective.
In arch specific code your fine to use what you think is a better fit. There's a proposal though to make these ELF annotations arch-independent, and if we went that route, we'd have to use % (or allow arch overrides) to fit all architectures.
Jan
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
In reply to Shawn AnastasioQuoted 7 lines · Shawn Anastasio
Quoted 6 lines · Andrew Cooper
Quoted 5 lines · Shawn Anastasio
Add the build system changes required to build for ppc64le (POWER8+). As of now the resulting image simply boots to an infinite loop.
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen build
Quoted 7 lines
Quoted 2 lines
I think the first of these isn't needed, given the config
ARCH_DEFCONFIGdefault. I'd suggest dropping it.It seems like the build system expects an
$(ARCH)_defconfigpresent if you don't manually specify a defconfig target. I see riscv64 has atiny64_defconfigand ariscv64_defconfigthat are idential, probably for this same reason. Would you like me to take the same approach of duplicatingopenpower_defconfigtoppc64_defconfig?
Or just rename openpower_defconfig to ppc64_defconfig ?
Is there any reason to keep openpower differently?
Quoted 6 lines
Quoted 3 lines
If that's not feasible, then fine, but if it is, it ought to be the default. Which might be as simple as saying "or later" somewhere in this text, or might be a giant can of worms that I shouldn't open...
Originally the help text for the two ISA config options ended in a "+" but that was deemed ambiguous. Would adding "or later" to the help text for the two options clarify it sufficiently?
Yeah, that would definitely help.
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/ppc/include/asm/page-bits.h new file +0
-1--- /dev/nullQuoted 8 lines
xen/arch/ppc/include/asm/page-bits.h line 5
Quoted 7 lines
+++
b/xen/arch/ppc/include/asm/page-bits.hxen/arch/ppc/include/asm/page-bits.h new file +5
-0@@ -0,0 +1,7 @@ +#ifndef __PPC_PAGE_BITS_H__ +#define __PPC_PAGE_BITS_H__ + +#define PAGE_SHIFT 16/* 64 KiB Pages */ +#define PADDR_BITS 48Is 64k the minimum granularity? Or is 4k an option?
Both 4K and 64K are supported by the hardware.
Quoted 4 lines
I ask because Xen has some very short sighted ABIs which we're still working on removing. There are still quite a few expectations of
PAGE_SHIFTbeing 12.
Quoted 6 lines
Quoted 3 lines
To be clear, we're looking to fix all of these ABIs, but I suspect it will be an easier lift to begin with at 4k. (Or perhaps the right thing is to double down and just get them fixed.)
Interesting. Given this I'm inclined to go with 4k just to reduce pain points during initial bring up, though supporting 64k would definitely be desirable going forward.
Maybe keep as 64k for now, with 4k as a backup if major problems are encountered?
I honestly don't know how much of Xen's common code is non-4k-clean, and this is the best opportunity to find out.
Quoted 54 lines
Quoted 48 lines
Quoted 35 lines
xen/arch/ppc/include/asm/page-bits.h new file +2
-0+ +#endif /* __PPC_PAGE_BITS_H__ */ xen/arch/ppc/ppc64/Makefile new file +1
-0@@ -0,0 +1 @@ +obj-y += head.oxen/arch/ppc/ppc64/asm-offsets.c new file +0
-0xen/arch/ppc/ppc64/head.S new file +17
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */ + tdi 0, 0, 0x48 /* Reverse endian of b . + 8 */ + b . + 44 /* Skip trampoline if endian is good */ + .long 0xa600607d /* mfmsr r11 */ + .long 0x01006b69 /* xori r11,r11,1 */ + .long 0x00004039 /* li r10,0 */xen/arch/ppc/ppc64/head.S line 27
Quoted 10 lines
xen/arch/ppc/ppc64/head.S new file +10
-0+ .long 0x6401417d /* mtmsrd r10,1 */ + .long 0x05009f42 /* bcl 20,31,$+4 */ + .long 0xa602487d /* mflr r10 */ + .long 0x14004a39 /* addi r10,r10,20 */ + .long 0xa6035a7d /* mtsrr0 r10 */ + .long 0xa6037b7d /* mtsrr1 r11 */ + .long 0x2400004c /* rfid */ + + /* Now that the endianness is confirmed, continue */ +1: b 1b.size start, . - start
.type start, %functionQuoted 1 line
Lets get the ELF metadata right from the start.
Good point. Following the example in the Power ELFv2 ABI specification [1] for function type annotation, I'll place
.type start, @function
Quoted 8 lines
in the ENTRY macro. It's not clear what the difference between %function and @function are in this context (my toolchain seems to accept both and produce the same ELF metadata), but the latter is more idiomatic in Power assembly. The same goes for its placement before the entrypoint vs. after the last instruction.
As for the size annotation, I'll follow Julien's suggestion and introduce an END macro.
There are reasons why this doesn't exist/work yet, and that's a swamp you swamp you absolutely don't want to wade into at this point.
Just leave the raw annotations for now, and they'll be cleaned up when the bugs with the various proposed options have been fixed.
~Andrew
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Hi,
Quoted 54 lines · Andrew Cooper
Quoted 7 lines · Shawn Anastasio
Quoted 6 lines · Andrew Cooper
Quoted 5 lines · Shawn Anastasio
Add the build system changes required to build for ppc64le (POWER8+). As of now the resulting image simply boots to an infinite loop.
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen buildQuoted 7 lines
Quoted 2 lines
I think the first of these isn't needed, given the config
ARCH_DEFCONFIGdefault. I'd suggest dropping it.It seems like the build system expects an
$(ARCH)_defconfigpresent if you don't manually specify a defconfig target. I see riscv64 has atiny64_defconfigand ariscv64_defconfigthat are idential, probably for this same reason. Would you like me to take the same approach of duplicatingopenpower_defconfigtoppc64_defconfig?Or just rename
openpower_defconfigtoppc64_defconfig?Is there any reason to keep openpower differently?
Quoted 6 lines
Quoted 3 lines
If that's not feasible, then fine, but if it is, it ought to be the default. Which might be as simple as saying "or later" somewhere in this text, or might be a giant can of worms that I shouldn't open...
Originally the help text for the two ISA config options ended in a "+" but that was deemed ambiguous. Would adding "or later" to the help text for the two options clarify it sufficiently?
Yeah, that would definitely help.
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/ppc/include/asm/page-bits.h new file +0
-1--- /dev/nullQuoted 8 lines
xen/arch/ppc/include/asm/page-bits.h line 5
Quoted 7 lines
+++
b/xen/arch/ppc/include/asm/page-bits.hxen/arch/ppc/include/asm/page-bits.h new file +5
-0@@ -0,0 +1,7 @@ +#ifndef __PPC_PAGE_BITS_H__ +#define __PPC_PAGE_BITS_H__ + +#define PAGE_SHIFT 16/* 64 KiB Pages */ +#define PADDR_BITS 48Is 64k the minimum granularity? Or is 4k an option?
Both 4K and 64K are supported by the hardware.
Quoted 4 lines
I ask because Xen has some very short sighted ABIs which we're still working on removing. There are still quite a few expectations of
PAGE_SHIFTbeing 12.Quoted 6 lines
Quoted 3 lines
To be clear, we're looking to fix all of these ABIs, but I suspect it will be an easier lift to begin with at 4k. (Or perhaps the right thing is to double down and just get them fixed.)
Interesting. Given this I'm inclined to go with 4k just to reduce pain points during initial bring up, though supporting 64k would definitely be desirable going forward.
Quoted 5 lines
Maybe keep as 64k for now, with 4k as a backup if major problems are encountered?
I honestly don't know how much of Xen's common code is non-4k-clean, and this is the best opportunity to find out.
The hypervisor itself is probably alright. For the tools and the kernel, we did some work a few years ago so the code don't assume the kernel and the hypervisor are using the same page size.
The tools and kernel have hardcoded expectation for the tools. Have a look at XC_PAGE_SIZE (tools) and XEN_PAGE_SHIFT (linux).
There was an attempt from Costin Lupu a couple of years ago to clean-up the definition (see [1]) but this was never merged. I can't remember why...
Now regarding the default page-size for PPC. At the moment, the page-size of the ABI is tie to the hypervisor.
For the ABI, it is best to use the bigger size (i.e. 64KB) because with just some rework in the hypervisor, you could run the same guest image on both a 4KB and 64KB.
Therefore, I think the 64KB size for the hypervisor is probably the better choice for the initial port. This will avoid some external ABI issue in the future if you want to support more page-size (we have the problem on Arm as we started with 4KB). You would also not rely on a newer ABI.
Cheers,
[1] https://lore.kernel.org/xen-devel/[email protected]/
Signature · 1 line
Julien Grall
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 80 lines · Julien Grall
Hi,
Quoted 54 lines · Andrew Cooper
Quoted 7 lines · Shawn Anastasio
Quoted 6 lines · Andrew Cooper
Quoted 5 lines · Shawn Anastasio
Add the build system changes required to build for ppc64le (POWER8+). As of now the resulting image simply boots to an infinite loop.
$ make XEN_TARGET_ARCH=ppc64 -C xen openpower_defconfig $ make XEN_TARGET_ARCH=ppc64 SUBSYSTEMS=xen -C xen buildQuoted 7 lines
Quoted 2 lines
I think the first of these isn't needed, given the config
ARCH_DEFCONFIGdefault. I'd suggest dropping it.It seems like the build system expects an
$(ARCH)_defconfigpresent if you don't manually specify a defconfig target. I see riscv64 has atiny64_defconfigand ariscv64_defconfigthat are idential, probably for this same reason. Would you like me to take the same approach of duplicatingopenpower_defconfigtoppc64_defconfig?Or just rename
openpower_defconfigtoppc64_defconfig?Is there any reason to keep openpower differently?
Quoted 6 lines
Quoted 3 lines
If that's not feasible, then fine, but if it is, it ought to be the default. Which might be as simple as saying "or later" somewhere in this text, or might be a giant can of worms that I shouldn't open...
Originally the help text for the two ISA config options ended in a "+" but that was deemed ambiguous. Would adding "or later" to the help text for the two options clarify it sufficiently?
Yeah, that would definitely help.
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/ppc/include/asm/page-bits.h new file +0
-1--- /dev/nullQuoted 8 lines
xen/arch/ppc/include/asm/page-bits.h line 5
Quoted 7 lines
+++
b/xen/arch/ppc/include/asm/page-bits.hxen/arch/ppc/include/asm/page-bits.h new file +5
-0@@ -0,0 +1,7 @@ +#ifndef __PPC_PAGE_BITS_H__ +#define __PPC_PAGE_BITS_H__ + +#define PAGE_SHIFT 16/* 64 KiB Pages */ +#define PADDR_BITS 48Is 64k the minimum granularity? Or is 4k an option?
Both 4K and 64K are supported by the hardware.
Quoted 4 lines
I ask because Xen has some very short sighted ABIs which we're still working on removing. There are still quite a few expectations of
PAGE_SHIFTbeing 12.Quoted 6 lines
Quoted 3 lines
To be clear, we're looking to fix all of these ABIs, but I suspect it will be an easier lift to begin with at 4k. (Or perhaps the right thing is to double down and just get them fixed.)
Interesting. Given this I'm inclined to go with 4k just to reduce pain points during initial bring up, though supporting 64k would definitely be desirable going forward.
Quoted 5 lines
Maybe keep as 64k for now, with 4k as a backup if major problems are encountered?
I honestly don't know how much of Xen's common code is non-4k-clean, and this is the best opportunity to find out.
The hypervisor itself is probably alright. For the tools and the kernel, we did some work a few years ago so the code don't assume the kernel and the hypervisor are using the same page size.
The tools and kernel have hardcoded expectation for the tools. Have a look at
XC_PAGE_SIZE(tools) andXEN_PAGE_SHIFT(linux).There was an attempt from Costin Lupu a couple of years ago to clean-up the definition (see [1]) but this was never merged. I can't remember why...
Now regarding the default page-size for PPC. At the moment, the page-size of the ABI is tie to the hypervisor.
For the ABI, it is best to use the bigger size (i.e. 64KB) because with just some rework in the hypervisor, you could run the same guest image on both a 4KB and 64KB.
Quoted 5 lines
Therefore, I think the 64KB size for the hypervisor is probably the better choice for the initial port. This will avoid some external ABI issue in the future if you want to support more page-size (we have the problem on Arm as we started with 4KB). You would also not rely on a newer ABI.
Just one remark here for consideration: Grant v2's sub-page grants use a 16-bit length and page offset. There would be a corner case which isn't representable (offset 0 length 64k).
Jan
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
In reply to Shawn AnastasioQuoted 8 lines · Shawn Anastasio
xen/arch/ppc/ppc64/head.S new file +2
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +
xen/arch/ppc/ppc64/head.S line 12
Quoted 10 lines
xen/arch/ppc/ppc64/head.S new file +10
-0+.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */
Sorry, I also meant to ask. How prevalent is Big Endian in practice in the Power world?
It's another area (like 4k pages) where I expect there to be plenty of fun to be had with the codebase.
~Andrew
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 2 lines · Andrew Cooper
Sorry, I also meant to ask. How prevalent is Big Endian in practice in the Power world?
Modern systems support operating in either endianness, but historically most operating systems targeted Big Endian-only, and some older systems didn't support Little Endian at all.
These days Little Endian is more prevalent (at least in the Open Source world), and many Linux distributions only target LE. Despite this though, most firmware on Power systems still operates in Big Endian mode exclusively,
so it's the responsibility of the kernel to handle the switch to LE at its entrypoint.
The FW being BE also needs to be considered whenever the kernel calls into firmware, since it's the responsibility of the kernel to switch to BE before making the call, and also to switch itself back to LE after the FW routine returns. Typically it's just handled by sprinkling this trampoline (via a macro) at all of the entry/return points.
Quoted 2 lines
It's another area (like 4k pages) where I expect there to be plenty of fun to be had with the codebase.
Hopefully the choice to run in Little Endian mode will reduce the number of pain points encountered.
Also it's worth mentioning that both 4k and 64k pages are supported, though 64k is probably the more common choice.
Quoted 1 line
~Andrew
Thanks,
Shawn
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
In reply to Andrew CooperQuoted 23 lines · Andrew Cooper
Quoted 8 lines · Shawn Anastasio
xen/arch/ppc/ppc64/head.S new file +2
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +xen/arch/ppc/ppc64/head.S line 12
Quoted 10 lines
xen/arch/ppc/ppc64/head.S new file +10
-0+.section .text.header, "ax", %progbits + +ENTRY(start) + /* + * Depending on how we were booted, the CPU could be running in either + * Little Endian or Big Endian mode. The following trampoline from Linux + * cleverly uses an instruction that encodes to a NOP if the CPU's + * endianness matches the assumption of the assembler (LE, in our case) + * or a branch to code that performs the endian switch in the other case. + */Sorry, I also meant to ask. How prevalent is Big Endian in practice in the Power world?
Quoted 2 lines
It's another area (like 4k pages) where I expect there to be plenty of fun to be had with the codebase.
Why? I'd expect "fun" if ppc was meaning to run in big-endian mode.
Jan
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
In reply to Shawn AnastasioQuoted 15 lines · Shawn Anastasio
xen/arch/ppc/Kconfig new file +12
-0@@ -0,0 +1,42 @@ +config PPC + def_bool y + +config PPC64 + def_bool y + select 64BIT + +config ARCH_DEFCONFIG + string + default "arch/ppc/configs/openpower_defconfig" + +menu "Architecture Features"
xen/arch/ppc/Kconfig line 22
Quoted 10 lines
xen/arch/ppc/Kconfig new file +10
-0+ +source "arch/Kconfig" + +endmenu + +menu "ISA Selection" + +choice + prompt "Base ISA" + default POWER_ISA_2_07B if PPC64
I think the "if" here is at best confusing. If / when ppc32 support is added, a potentially different default here would make necessary adjustments, yet imo would not be very likely to introduce this very "if".
xen/arch/ppc/arch.mk line 8
Quoted 11 lines
xen/arch/ppc/arch.mk new file +8
-0@@ -0,0 +1,12 @@ +######################################## +# Power-specific definitions + +ppc-march-$(CONFIG_POWER_ISA_2_07B) := power8 +ppc-march-$(CONFIG_POWER_ISA_3_00) := power9 + +CFLAGS += -m64 -mlittle-endian -mcpu=$(ppc-march-y) +CFLAGS += -mstrict-align -mcmodel=large -mabi=elfv2 -mno-altivec -mno-vsx
Just for my own education: Besides the expected effect, -mstrict-align also appears to imply -mbit-align, which I'm not sure is intended here. Could you clarify the intentions for me?
As to -mabi=elfv2, it looks as if this limits us to gcc12 and newer. That's fine, but I think it wants pointing out in ./README (which has a section for this kind of information).
Quoted 27 lines
xen/arch/ppc/include/asm/config.h new file +24
-0@@ -0,0 +1,63 @@ +#ifndef __PPC_CONFIG_H__ +#define __PPC_CONFIG_H__ + +#include <xen/const.h> +#include <xen/page-size.h> + +#if defined(CONFIG_PPC64 ) +#define LONG_BYTEORDER 3 +#define ELFSIZE 64 +#define MAX_VIRT_CPUS 1024u +#else +#error "Unsupported PowerPC variant" +#endif + +#define BYTES_PER_LONG (1 << LONG_BYTEORDER) +#define BITS_PER_LONG (BYTES_PER_LONG << 3) +#define POINTER_ALIGN BYTES_PER_LONG + +#define BITS_PER_LLONG 64 + +/* xen_ulong_t is always 64 bits */ +#define BITS_PER_XEN_ULONG 64 + +#define CONFIG_PPC_L1_CACHE_SHIFT 7
xen/arch/ppc/include/asm/config.h line 34
Quoted 10 lines
xen/arch/ppc/include/asm/config.h new file +10
-0+#define CONFIG_PAGEALLOC_MAX_ORDER 18 +#define CONFIG_DOMU_MAX_ORDER 9 +#define CONFIG_HWDOM_MAX_ORDER 10 + +#define OPT_CONSOLE_STR "dtuart" +#define INVALID_VCPU_ID MAX_VIRT_CPUS + +/* Linkage for PPC */ +#ifdef __ASSEMBLY__ +#define ALIGN .align 2
I think I did ask for the same on RISC-V (yet sadly it's still .align there): .align is notoriously ambiguous. May I ask that you use .p2align (which I think is what is meant here, else .balign)?
Jan
Re: [PATCH v4 2/4] xen: Add files needed for minimal ppc64le build
Quoted 23 lines · Jan Beulich
Quoted 22 lines · Shawn Anastasio
xen/arch/ppc/Kconfig new file +19
-0@@ -0,0 +1,42 @@ +config PPC + def_bool y + +config PPC64 + def_bool y + select 64BIT + +config ARCH_DEFCONFIG + string + default "arch/ppc/configs/openpower_defconfig" + +menu "Architecture Features" + +source "arch/Kconfig" + +endmenu + +menu "ISA Selection" +
Quoted 8 lines
xen/arch/ppc/Kconfig line 22
Quoted 3 lines
xen/arch/ppc/Kconfig new file +3
-0+choice + prompt "Base ISA" + default POWER_ISA_2_07B if PPC64I think the "if" here is at best confusing. If / when ppc32 support is added, a potentially different default here would make necessary adjustments, yet imo would not be very likely to introduce this very "if".
Fair enough, I'll drop the if.
Quoted 7 lines
Quoted 7 lines
xen/arch/ppc/arch.mk new file +4
-0@@ -0,0 +1,12 @@ +######################################## +# Power-specific definitions + +ppc-march-$(CONFIG_POWER_ISA_2_07B) := power8
Quoted 8 lines
xen/arch/ppc/arch.mk line 8
Quoted 4 lines
xen/arch/ppc/arch.mk new file +4
-0+ppc-march-$(CONFIG_POWER_ISA_3_00) := power9 + +CFLAGS += -m64 -mlittle-endian -mcpu=$(ppc-march-y) +CFLAGS += -mstrict-align -mcmodel=large -mabi=elfv2 -mno-altivec -mno-vsxJust for my own education: Besides the expected effect, -mstrict-align also appears to imply -mbit-align, which I'm not sure is intended here. Could you clarify the intentions for me?
If I'm reading the GCC documentation right, -mbit-align is the default behavior regardless of -mstrict-align. The intention was simply to avoid the generation of unaligned memory accesses in the kernel, since they are only handled in hardware up to a certain width.
Quoted 3 lines
As to -mabi=elfv2, it looks as if this limits us to gcc12 and newer. That's fine, but I think it wants pointing out in ./README (which has a section for this kind of information).
This is not the case. The ELFv2 ABI was released in 2014 and has been the de facto standard on ppc64le since its inception. I know that ELFv2 support in GCC dates back to at least version 6 (the earliest I can remember using), but I believe it goes back even further than that.
Quoted 33 lines
Quoted 33 lines
xen/arch/ppc/include/asm/config.h new file +30
-0@@ -0,0 +1,63 @@ +#ifndef __PPC_CONFIG_H__ +#define __PPC_CONFIG_H__ + +#include <xen/const.h> +#include <xen/page-size.h> + +#if defined(CONFIG_PPC64 ) +#define LONG_BYTEORDER 3 +#define ELFSIZE 64 +#define MAX_VIRT_CPUS 1024u +#else +#error "Unsupported PowerPC variant" +#endif + +#define BYTES_PER_LONG (1 << LONG_BYTEORDER) +#define BITS_PER_LONG (BYTES_PER_LONG << 3) +#define POINTER_ALIGN BYTES_PER_LONG + +#define BITS_PER_LLONG 64 + +/* xen_ulong_t is always 64 bits */ +#define BITS_PER_XEN_ULONG 64 + +#define CONFIG_PPC_L1_CACHE_SHIFT 7 +#define CONFIG_PAGEALLOC_MAX_ORDER 18 +#define CONFIG_DOMU_MAX_ORDER 9 +#define CONFIG_HWDOM_MAX_ORDER 10 + +#define OPT_CONSOLE_STR "dtuart" +#define INVALID_VCPU_ID MAX_VIRT_CPUS
Quoted 8 lines
xen/arch/ppc/include/asm/config.h line 34
Quoted 4 lines
xen/arch/ppc/include/asm/config.h new file +4
-0+ +/* Linkage for PPC */ +#ifdef __ASSEMBLY__ +#define ALIGN .align 2I think I did ask for the same on RISC-V (yet sadly it's still .align there): .align is notoriously ambiguous. May I ask that you use .p2align (which I think is what is meant here, else .balign)?
Sure, will do.
Quoted 1 line
Jan
Thanks,
Shawn
PATCH v4 3/4 automation: Add ppc64le cross-build jobs
In reply to Shawn AnastasioAdd build jobs to cross-compile Xen for ppc64le.
--- automation/gitlab-ci/build.yaml | 60 +++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)
automation/gitlab-ci/build.yaml +60 -0
@@ -183,6 +183,33 @@ variables: <<: *gcc +.ppc64le-cross-build-tmpl: + <<: *build + variables: + XEN_TARGET_ARCH: ppc64 + tags: + - x86_64 + +.ppc64le-cross-build: + extends: .ppc64le-cross-build-tmpl + variables: + debug: n + +.ppc64le-cross-build-debug: + extends: .ppc64le-cross-build-tmpl + variables: + debug: y + +.gcc-ppc64le-cross-build: + extends: .ppc64le-cross-build + variables: + <<: *gcc + +.gcc-ppc64le-cross-build-debug: + extends: .ppc64le-cross-build-debug + variables: + <<: *gcc + .yocto-test: stage: build image: registry.gitlab.com/xen-project/xen/${CONTAINER}
@@ -516,6 +543,39 @@ archlinux-current-gcc-riscv64-debug-randconfig: EXTRA_FIXED_RANDCONFIG: CONFIG_COVERAGE=n +# Power cross-build +debian-bullseye-gcc-ppc64le: + extends: .gcc-ppc64le-cross-build + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + HYPERVISOR_ONLY: y + +debian-bullseye-gcc-ppc64le-debug: + extends: .gcc-ppc64le-cross-build-debug + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + HYPERVISOR_ONLY: y + +debian-bullseye-gcc-ppc64le-randconfig: + extends: .gcc-ppc64le-cross-build + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + RANDCONFIG: y + EXTRA_FIXED_RANDCONFIG: + CONFIG_COVERAGE=n + +debian-bullseye-gcc-ppc64le-debug-randconfig: + extends: .gcc-ppc64le-cross-build-debug + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + RANDCONFIG: y + EXTRA_FIXED_RANDCONFIG: + CONFIG_COVERAGE=n + # Yocto test jobs yocto-qemuarm64: extends: .yocto-test-arm64
Signature · 1 line
2.30.2
Re: [PATCH v4 3/4] automation: Add ppc64le cross-build jobs
Quoted 3 lines · Shawn Anastasio
Add build jobs to cross-compile Xen for ppc64le.
Signed-off-by Shawn Anastasio <[email protected]>
Quoted 85 lines
--- automation/gitlab-ci/build.yaml | 60 +++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)automation/gitlab-ci/build.yaml +60
-0@@ -183,6 +183,33 @@ variables: <<: *gcc +.ppc64le-cross-build-tmpl: + <<: *build + variables: + XEN_TARGET_ARCH: ppc64 + tags: + - x86_64 + +.ppc64le-cross-build: + extends: .ppc64le-cross-build-tmpl + variables: + debug: n + +.ppc64le-cross-build-debug: + extends: .ppc64le-cross-build-tmpl + variables: + debug: y + +.gcc-ppc64le-cross-build: + extends: .ppc64le-cross-build + variables: + <<: *gcc + +.gcc-ppc64le-cross-build-debug: + extends: .ppc64le-cross-build-debug + variables: + <<: *gcc + .yocto-test: stage: build image: registry.gitlab.com/xen-project/xen/${CONTAINER}@@ -516,6 +543,39 @@ archlinux-current-gcc-riscv64-debug-randconfig: EXTRA_FIXED_RANDCONFIG: CONFIG_COVERAGE=n +# Power cross-build +debian-bullseye-gcc-ppc64le: + extends: .gcc-ppc64le-cross-build + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + HYPERVISOR_ONLY: y + +debian-bullseye-gcc-ppc64le-debug: + extends: .gcc-ppc64le-cross-build-debug + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + HYPERVISOR_ONLY: y + +debian-bullseye-gcc-ppc64le-randconfig: + extends: .gcc-ppc64le-cross-build + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + RANDCONFIG: y + EXTRA_FIXED_RANDCONFIG: + CONFIG_COVERAGE=n + +debian-bullseye-gcc-ppc64le-debug-randconfig: + extends: .gcc-ppc64le-cross-build-debug + variables: + CONTAINER: debian:bullseye-ppc64le + KBUILD_DEFCONFIG: openpower_defconfig + RANDCONFIG: y + EXTRA_FIXED_RANDCONFIG: + CONFIG_COVERAGE=n + # Yocto test jobs yocto-qemuarm64: extends: .yocto-test-arm64Signature · 1 line
2.30.2
Re: [PATCH v4 3/4] automation: Add ppc64le cross-build jobs
In reply to Shawn AnastasioQuoted 3 lines · Shawn Anastasio
Add build jobs to cross-compile Xen for ppc64le.
Signed-off-by Shawn Anastasio <[email protected]>
PATCH v4 4/4 maintainers: Add ppc64 maintainer
In reply to Shawn Anastasio--- MAINTAINERS | 4 ++++ 1 file changed, 4 insertions(+)
MAINTAINERS +4 -0
@@ -460,6 +460,10 @@ X: xen/arch/x86/acpi/lib.c F: xen/drivers/cpufreq/ F: xen/include/acpi/cpufreq/ +PPC64 +M: Shawn Anastasio <[email protected]> +F: xen/arch/ppc/ + PUBLIC I/O INTERFACES AND PV DRIVERS DESIGNS M: Juergen Gross <[email protected]> S: Supported
Signature · 1 line
2.30.2