Skip to contentSkip to conversation

tests/qtest/migration: Fix A-B file build

PATCH In review Fabiano Rosas 3 participants 6 messages Apr 29 – May 5

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/qtest/migration: Fix A-B file build

The generation of the guest workloads for migration-test is all broken:

1) We moved code around and forgot to update the includes from the guest workloads at qtest/migration/<ARCH>/.

2) Since this code doesn't build by default due to the need for cross compilation, we also broke the s390x code entirely by removing the libc implementation from sclp.c.

3) The build adds a blank line at the end of the header files due to how the sed command used to remove extra variable declarations is written.

4) Must be too long since we last built this thing that powerpc compilers would output BE by default. Nowadays we need -mbig-endian to force it.

5) Format of comments needs fixing to please checkpatch.

Fix everything. Most changes are trivial, except for s390x where I had to reimplement some of the routines from sclp.c. Compiling that object file in doesn't work because of its various includes that this standalone code can't provide.

Testing this change (assuming x86 host):

$ cd build
$ pushd tests/qtest/migration
$ make clean
$ make i386
$ make CROSS_PREFIX=your-favorite-cross-compiler- s390x
$ make CROSS_PREFIX=your-favorite-cross-compiler- aarch64
$ make CROSS_PREFIX=your-favorite-cross-compiler- ppc64
$ popd
$ make -j$(nproc) # must build the tests again
$ QTEST_QEMU_BINARY=./qemu-system-<ARCH> migration-test
Fixes 9f4278837d ("pc-bios/s390-ccw: Use the libc from SLOF and remove sclp prints")
Fixes e1803dabdc ("tests/qtest/migration: Move common test code")
---
 tests/qtest/migration/Makefile             |  11 +-
 tests/qtest/migration/aarch64/Makefile     |   4 +-
 tests/qtest/migration/aarch64/a-b-kernel.S |   2 +-
 tests/qtest/migration/aarch64/a-b-kernel.h |   7 +-
 tests/qtest/migration/bootfile.h           |  11 +
 tests/qtest/migration/i386/Makefile        |   4 +-
 tests/qtest/migration/i386/a-b-bootblock.S |   2 +-
 tests/qtest/migration/i386/a-b-bootblock.h |   8 +-
 tests/qtest/migration/ppc64/Makefile       |   6 +-
 tests/qtest/migration/ppc64/a-b-kernel.S   |   2 +-
 tests/qtest/migration/ppc64/a-b-kernel.h   |   8 +-
 tests/qtest/migration/s390x/Makefile       |   8 +-
 tests/qtest/migration/s390x/a-b-bios.c     |  66 ++-
 tests/qtest/migration/s390x/a-b-bios.h     | 609 +++++++++++++--------
 14 files changed, 491 insertions(+), 257 deletions(-)
tests/qtest/migration/Makefile +6 -5
index 2c5ee287ec..c183b69941 100644
@@ -7,7 +7,7 @@
 
 TARGET_LIST = i386 aarch64 s390x ppc64
 
-SRC_PATH = ../..
+SRC_PATH = ../../../..
 
 .PHONY: help $(TARGET_LIST)
 help:
@@ -23,16 +23,17 @@ help:
 	@echo " Possible targets are: $(TARGET_LIST)"
 
 override define __note
-/* This file is automatically generated from the assembly file in
- * tests/migration/$@. Edit that file and then run "make all"
- * inside tests/migration to update, and then remember to send both
+/*
+ * This file is automatically generated from the assembly file in
+ * tests/qtest/migration/$@. Edit that file and then run "make"
+ * inside tests/qtest/migration to update, and then remember to send both
  * the header and the assembler differences in your patch submission.
  */
 endef
 export __note
 
 $(TARGET_LIST):
-	$(MAKE) CROSS_PREFIX=$(CROSS_PREFIX) -C $@
+	$(MAKE) SRC_PATH=$(SRC_PATH) CROSS_PREFIX=$(CROSS_PREFIX) -C $@
 
 clean:
 	for target in $(TARGET_LIST); do \
tests/qtest/migration/aarch64/Makefile +2 -2
index 9c4fa18e76..fd102ab776 100644
@@ -6,7 +6,7 @@ all: a-b-kernel.h
 
 a-b-kernel.h: aarch64.kernel
 	echo "$$__note" > $@
-	xxd -i $< | sed -e 's/.*int.*//' >> $@
+	xxd -i $< | sed -e '/.*int.*/d' >> $@
 
 aarch64.kernel: aarch64.elf
 	$(CROSS_PREFIX)objcopy -O binary $< $@
@@ -15,4 +15,4 @@ aarch64.elf: a-b-kernel.S
 	$(CROSS_PREFIX)gcc -o $@ -nostdlib -Wl,--build-id=none $<
 
 clean:
-	$(RM) *.kernel *.elf
+	$(RM) *.kernel *.elf *.h
tests/qtest/migration/aarch64/a-b-kernel.S +1 -1
index a4103ecb71..8f29efbb4a 100644
@@ -11,7 +11,7 @@
 # pc-relative address. Also the branch instructions should use relative
 # addresses only.
 
-#include "../migration-test.h"
+#include "../bootfile.h"
 
 .section .text
 
tests/qtest/migration/aarch64/a-b-kernel.h +4 -3
index 34e518d061..c444cdaf73 100644
@@ -1,6 +1,7 @@
-/* This file is automatically generated from the assembly file in
- * tests/migration/aarch64. Edit that file and then run "make all"
- * inside tests/migration to update, and then remember to send both
+/*
+ * This file is automatically generated from the assembly file in
+ * tests/qtest/migration/aarch64. Edit that file and then run "make"
+ * inside tests/qtest/migration to update, and then remember to send both
  * the header and the assembler differences in your patch submission.
  */
 unsigned char aarch64_kernel[] = {
tests/qtest/migration/bootfile.h +11 -0
index 96e784b163..0ce5b34433 100644
@@ -8,6 +8,15 @@
 #ifndef BOOTFILE_H
 #define BOOTFILE_H
 
+/*
+ * This file is included from qtest, but also from the assembly code
+ * that generates the header file containing the guest workload. Don't
+ * expose the function declarations and non-standard types to it.
+ */
+#ifdef __ASSEMBLER__
+#define MIGRATION_GUEST_CODE
+#endif
+
 /* Common */
 #define TEST_MEM_PAGE_SIZE 4096
 
@@ -33,8 +42,10 @@
  */
 #define ARM_TEST_MAX_KERNEL_SIZE (512 * 1024)
 
+#ifndef MIGRATION_GUEST_CODE
 void bootfile_delete(void);
 char *bootfile_create(const char *arch, const char *dir, bool suspend_me);
 char *bootfile_get(void);
+#endif
 
 #endif /* BOOTFILE_H */
tests/qtest/migration/i386/Makefile +2 -2
index 37a72ae353..cbf63103af 100644
@@ -6,7 +6,7 @@ all: a-b-bootblock.h
 
 a-b-bootblock.h: x86.bootsect x86.o
 	echo "$$__note" > header.tmp
-	xxd -i $< | sed -e 's/.*int.*//' >> header.tmp
+	xxd -i $< | sed -e '/.*int.*/d' >> header.tmp
 	nm x86.o | awk '{print "#define SYM_"$$3" 0x"$$1}' >> header.tmp
 	mv header.tmp $@
 
@@ -20,4 +20,4 @@ x86.o: a-b-bootblock.S
 	$(CROSS_PREFIX)gcc -I.. -m32 -march=i486 -c $< -o $@
 
 clean:
-	@rm -rf *.boot *.o *.bootsect
+	@rm -rf *.boot *.o *.bootsect *.h
tests/qtest/migration/i386/a-b-bootblock.S +1 -1
index 6f39eb6051..99253f5ee5 100644
@@ -9,7 +9,7 @@
 #
 # Author: [email protected]
 
-#include "migration-test.h"
+#include "../bootfile.h"
 
 #define ACPI_ENABLE         0xf1
 #define ACPI_PORT_SMI_CMD   0xb2
tests/qtest/migration/i386/a-b-bootblock.h +4 -4
index c83f8711db..3b5de60161 100644
@@ -1,6 +1,7 @@
-/* This file is automatically generated from the assembly file in
- * tests/migration/i386. Edit that file and then run "make all"
- * inside tests/migration to update, and then remember to send both
+/*
+ * This file is automatically generated from the assembly file in
+ * tests/qtest/migration/i386. Edit that file and then run "make"
+ * inside tests/qtest/migration to update, and then remember to send both
  * the header and the assembler differences in your patch submission.
  */
 unsigned char x86_bootsect[] = {
@@ -48,7 +49,6 @@ unsigned char x86_bootsect[] = {
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x55, 0xaa
 };
-
 #define SYM_do_zero 0x00007c3d
 #define SYM_gdt 0x00007ca0
 #define SYM_gdtdesc 0x00007cb8
tests/qtest/migration/ppc64/Makefile +3 -3
index a3a2d98ac8..1bd6d35799 100644
@@ -3,13 +3,13 @@ all: a-b-kernel.h
 
 a-b-kernel.h: ppc64.kernel
 	echo "$$__note" > $@
-	xxd -i $< | sed -e 's/.*int.*//' >> $@
+	xxd -i $< | sed -e '/.*int.*/d' >> $@
 
 ppc64.kernel: ppc64.elf
 	$(CROSS_PREFIX)objcopy -O binary -S $< $@
 
 ppc64.elf: a-b-kernel.S
-	$(CROSS_PREFIX)gcc -static -o $@ -nostdlib -Wl,--build-id=none $<
+	$(CROSS_PREFIX)gcc -mbig-endian -static -o $@ -nostdlib -Wl,--build-id=none $<
 
 clean:
-	$(RM) *.kernel *.elf
+	$(RM) *.kernel *.elf *.h
tests/qtest/migration/ppc64/a-b-kernel.S +1 -1
index 0613a8d18e..c0009ecd10 100644
@@ -4,7 +4,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
-#include "../migration-test.h"
+#include "../bootfile.h"
 
 .section .text
 
tests/qtest/migration/ppc64/a-b-kernel.h +4 -4
index 673317efdb..8dc2bcd61a 100644
@@ -1,6 +1,7 @@
-/* This file is automatically generated from the assembly file in
- * tests/migration/ppc64. Edit that file and then run "make all"
- * inside tests/migration to update, and then remember to send both
+/*
+ * This file is automatically generated from the assembly file in
+ * tests/qtest/migration/ppc64. Edit that file and then run "make"
+ * inside tests/qtest/migration to update, and then remember to send both
  * the header and the assembler differences in your patch submission.
  */
 unsigned char ppc64_kernel[] = {
@@ -39,4 +40,3 @@ unsigned char ppc64_kernel[] = {
   0x38, 0xa0, 0x00, 0x01, 0x38, 0xc0, 0x00, 0x42, 0x78, 0xc6, 0xc1, 0xc6,
   0x44, 0x00, 0x00, 0x22, 0x4b, 0xff, 0xff, 0xcc
 };
-
tests/qtest/migration/s390x/Makefile +4 -4
index 6671de2efc..a2399821a7 100644
@@ -3,7 +3,7 @@
 
 .PHONY: all clean
 all: a-b-bios.h
-fwdir=../../../pc-bios/s390-ccw
+fwdir=$(SRC_PATH)/pc-bios/s390-ccw
 
 CFLAGS+=-ffreestanding -fno-delete-null-pointer-checks -fPIE -Os \
 	-msoft-float -march=z900 -fno-asynchronous-unwind-tables \
@@ -11,14 +11,14 @@ CFLAGS+=-ffreestanding -fno-delete-null-pointer-checks -fPIE -Os \
 
 a-b-bios.h: s390x.elf
 	echo "$$__note" > header.tmp
-	xxd -i $< | sed -e 's/.*int.*//' >> header.tmp
+	xxd -i $< | sed -e '/.*int.*/d' >> header.tmp
 	mv header.tmp $@
 
 # We use common-page-size=16 to avoid big padding in the ELF file
 s390x.elf: a-b-bios.c
 	$(CROSS_PREFIX)gcc $(CFLAGS) -I$(fwdir) $(fwdir)/start.S \
-		$(fwdir)/sclp.c -Wl,-zcommon-page-size=16 -o $@ $<
+	 -Wl,-zcommon-page-size=16 -o $@ $<
 	$(CROSS_PREFIX)strip $@
 
 clean:
-	@rm -rf *.elf *.o
+	@rm -rf *.elf *.o *.h
tests/qtest/migration/s390x/a-b-bios.c +59 -7

88 lines not on this page · raw message

tests/qtest/migration/s390x/a-b-bios.h +389 -220

662 lines not on this page · raw message

Signature · 1 line
2.51.0
#1 lore raw
Quoted 37 lines · Fabiano Rosas

The generation of the guest workloads for migration-test is all broken:

1) We moved code around and forgot to update the includes from the guest workloads at qtest/migration/<ARCH>/.

2) Since this code doesn't build by default due to the need for cross compilation, we also broke the s390x code entirely by removing the libc implementation from sclp.c.

3) The build adds a blank line at the end of the header files due to how the sed command used to remove extra variable declarations is written.

4) Must be too long since we last built this thing that powerpc compilers would output BE by default. Nowadays we need -mbig-endian to force it.

5) Format of comments needs fixing to please checkpatch.

Fix everything. Most changes are trivial, except for s390x where I had to reimplement some of the routines from sclp.c. Compiling that object file in doesn't work because of its various includes that this standalone code can't provide.

Testing this change (assuming x86 host):

$ cd build
$ pushd tests/qtest/migration
$ make clean
$ make i386
$ make CROSS_PREFIX=your-favorite-cross-compiler- s390x
$ make CROSS_PREFIX=your-favorite-cross-compiler- aarch64
$ make CROSS_PREFIX=your-favorite-cross-compiler- ppc64
$ popd
$ make -j$(nproc) # must build the tests again
$ QTEST_QEMU_BINARY=./qemu-system-<ARCH> migration-test
Quoted 3 lines
Fixes 9f4278837d ("pc-bios/s390-ccw: Use the libc from SLOF and remove sclp prints")
Fixes e1803dabdc ("tests/qtest/migration: Move common test code")

Since things are essentially broken, I queued this one directly. However I don't have any knowledge to review the s390 changes, please s390x developers do so and if things need change we can do follow ups.

I did test all these and they work. One thing to mention below,

Quoted 7 lines
---
 tests/qtest/migration/Makefile             |  11 +-
 tests/qtest/migration/aarch64/Makefile     |   4 +-
 tests/qtest/migration/aarch64/a-b-kernel.S |   2 +-
 tests/qtest/migration/aarch64/a-b-kernel.h |   7 +-
 tests/qtest/migration/bootfile.h           |  11 +
 tests/qtest/migration/i386/Makefile        |   4 +-
Quoted 8 lines
 tests/qtest/migration/i386/a-b-bootblock.S |   2 +-
 tests/qtest/migration/i386/a-b-bootblock.h |   8 +-
 tests/qtest/migration/ppc64/Makefile       |   6 +-
 tests/qtest/migration/ppc64/a-b-kernel.S   |   2 +-
 tests/qtest/migration/ppc64/a-b-kernel.h   |   8 +-
 tests/qtest/migration/s390x/Makefile       |   8 +-
 tests/qtest/migration/s390x/a-b-bios.c     |  66 ++-
 tests/qtest/migration/s390x/a-b-bios.h     | 609 +++++++++++++--------

I somehow generated different a-b-bios.h for s390x after cross-make. But I verified both versions of s390x_elf[] work here.

My gut feeling is it is a compiler difference causing different code generated, even if there's no real functional difference.

My "your-favorite-cross-compiler-" dumps this:

$ s390x-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/s390x-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/s390x-linux-gnu/15/lto-wrapper
Target: s390x-linux-gnu
Configured with: ../gcc-15.2.1-20250808/configure --bindir=/usr/bin --build=x86_64-redhat-linux-gnu --datadir=/usr/share --disable-decimal-float --disable-dependency-tracking --disable-gold --disable-libgcj --disable-libgomp --disable-libmpx --disable-libquadmath --disable-libssp --disable-libunwind-exceptions --disable-shared --disable-silent-rules --disable-sjlj-exceptions --disable-threads --with-ld=/usr/bin/s390x-linux-gnu-ld --enable-__cxa_atexit --enable-checking=release --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++ --enable-linker-build-id --enable-lto --enable-nls --enable-obsolete --enable-plugin --enable-targets=all --exec-prefix=/usr --host=x86_64-redhat-linux-gnu --includedir=/usr/include --infodir=/usr/share/info --libexecdir=/usr/libexec --localstatedir=/var --mandir=/usr/share/man --prefix=/usr --program-prefix=s390x-linux-gnu- --sbindir=/usr/bin --sharedstatedir=/var/lib --sysconfdir=/etc --target=s390x-linux-gnu --with-bugurl=http://bugzilla.redhat.com/bugzilla/ --with-gcc-major-version-only --with-isl --with-newlib --with-plugin-ld=/usr/bin/s390x-linux-gnu-ld --with-sysroot=/usr/s390x-linux-gnu/sys-root --with-system-libunwind --with-system-zlib --without-headers --with-arch=zEC12 --with-tune=z13 --enable-decimal-float --with-long-double-128 --enable-gnu-indirect-function --with-linker-hash-style=gnu
Thread model: single
Supported LTO compression algorithms: zlib zstd
gcc version 15.2.1 20250808 (Red Hat Cross 15.2.1-1) (GCC) 
Signature · 1 line
Peter Xu
#2 lore raw
Quoted 38 lines · Peter Xu
Quoted 37 lines · Fabiano Rosas

The generation of the guest workloads for migration-test is all broken:

1) We moved code around and forgot to update the includes from the guest workloads at qtest/migration/<ARCH>/.

2) Since this code doesn't build by default due to the need for cross compilation, we also broke the s390x code entirely by removing the libc implementation from sclp.c.

3) The build adds a blank line at the end of the header files due to how the sed command used to remove extra variable declarations is written.

4) Must be too long since we last built this thing that powerpc compilers would output BE by default. Nowadays we need -mbig-endian to force it.

5) Format of comments needs fixing to please checkpatch.

Fix everything. Most changes are trivial, except for s390x where I had to reimplement some of the routines from sclp.c. Compiling that object file in doesn't work because of its various includes that this standalone code can't provide.

Testing this change (assuming x86 host):

$ cd build
$ pushd tests/qtest/migration
$ make clean
$ make i386
$ make CROSS_PREFIX=your-favorite-cross-compiler- s390x
$ make CROSS_PREFIX=your-favorite-cross-compiler- aarch64
$ make CROSS_PREFIX=your-favorite-cross-compiler- ppc64
$ popd
$ make -j$(nproc) # must build the tests again
$ QTEST_QEMU_BINARY=./qemu-system-<ARCH> migration-test
Quoted 7 lines
Quoted 3 lines
Fixes 9f4278837d ("pc-bios/s390-ccw: Use the libc from SLOF and remove sclp prints")
Fixes e1803dabdc ("tests/qtest/migration: Move common test code")

Since things are essentially broken, I queued this one directly. However I don't have any knowledge to review the s390 changes, please s390x developers do so and if things need change we can do follow ups.

Thanks, this fell off my list. I did at least test on real s390 hardware and cross compile for i386/aarch64/ppc64 to verify 1) yep, s390x was broken before this patch and 2) I can test s390x successfully (as well as cross-compiled i386/aarch64/ppc64 on s390)

I did look at the sclp re-implementation, that code looks sane.

@Jared as a follow-up can you look at the possibility of cleaning it up so that we can go back to using shared routines for this test?

[...]

Quoted 5 lines

I somehow generated different a-b-bios.h for s390x after cross-make. But I verified both versions of s390x_elf[] work here.

My gut feeling is it is a compiler difference causing different code generated, even if there's no real functional difference.

I also see a difference between the a-b-bios.h generated, and also verified the s390 test passes either way. My assumption is the same as yours. In my case I was using

gcc (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)

Thanks,
Matt

#3 lore raw
In reply to Fabiano Rosas
Quoted 6 lines · Fabiano Rosas
tests/qtest/migration/bootfile.h +0 -0
index 96e784b163..0ce5b34433 100644
@@ -8,6 +8,15 @@
 #ifndef BOOTFILE_H

tests/qtest/migration/bootfile.h line 18

Quoted 10 lines
tests/qtest/migration/bootfile.h +8 -0
 #define BOOTFILE_H
 
+/*
+ * This file is included from qtest, but also from the assembly code
+ * that generates the header file containing the guest workload. Don't
+ * expose the function declarations and non-standard types to it.
+ */
+#ifdef __ASSEMBLER__
+#define MIGRATION_GUEST_CODE
+#endif

Just to mention: this triggers a checkpatch warning (I only notice this when running checkpatch locally; CI is fine with warnings), which seems to be false positive.

But still, I wonder whether we need this at all. IIUC we can also put those into a separate smaller header and include that here, then use that one for arch a-b-bios builds.

Even if it will work, we can leave it for later.

Quoted 13 lines
tests/qtest/migration/bootfile.h +3 -0
+
 /* Common */
 #define TEST_MEM_PAGE_SIZE 4096
 
@@ -33,8 +42,10 @@
  */
 #define ARM_TEST_MAX_KERNEL_SIZE (512 * 1024)
 
+#ifndef MIGRATION_GUEST_CODE
 void bootfile_delete(void);
 char *bootfile_create(const char *arch, const char *dir, bool suspend_me);
 char *bootfile_get(void);
+#endif
Signature · 1 line
Peter Xu
#4 lore raw
Quoted 22 lines · Peter Xu
Quoted 6 lines · Fabiano Rosas
tests/qtest/migration/bootfile.h +0 -0
index 96e784b163..0ce5b34433 100644
@@ -8,6 +8,15 @@
 #ifndef BOOTFILE_H

tests/qtest/migration/bootfile.h line 18

Quoted 10 lines
tests/qtest/migration/bootfile.h +8 -0
 #define BOOTFILE_H
 
+/*
+ * This file is included from qtest, but also from the assembly code
+ * that generates the header file containing the guest workload. Don't
+ * expose the function declarations and non-standard types to it.
+ */
+#ifdef __ASSEMBLER__
+#define MIGRATION_GUEST_CODE
+#endif

Just to mention: this triggers a checkpatch warning (I only notice this when running checkpatch locally; CI is fine with warnings), which seems to be false positive.

Quoted 6 lines

But still, I wonder whether we need this at all. IIUC we can also put those into a separate smaller header and include that here, then use that one for arch a-b-bios builds.

Even if it will work, we can leave it for later.

I have a bunch of patches adding code into bootfile.c and consequently bootfile.h for the stress.c and hugepages work I told you about. We can discuss the header structure when I send them, if you don't mind.

Quoted 13 lines
Quoted 13 lines
tests/qtest/migration/bootfile.h +3 -0
+
 /* Common */
 #define TEST_MEM_PAGE_SIZE 4096
 
@@ -33,8 +42,10 @@
  */
 #define ARM_TEST_MAX_KERNEL_SIZE (512 * 1024)
 
+#ifndef MIGRATION_GUEST_CODE
 void bootfile_delete(void);
 char *bootfile_create(const char *arch, const char *dir, bool suspend_me);
 char *bootfile_get(void);
+#endif
#5 lore raw
Quoted 3 lines · Fabiano Rosas

I have a bunch of patches adding code into bootfile.c and consequently bootfile.h for the stress.c and hugepages work I told you about. We can discuss the header structure when I send them, if you don't mind.

Sure thing.

Signature · 1 line
Peter Xu