Skip to contentSkip to conversation

selftests/nolibc: add minimal kernel config support - part1

PATCH In review Zhangjin Wu 3 participants 26 messages Jul 27, 2023 – Jul 30, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover selftests/nolibc: add minimal kernel config support - part1

Hi, Willy

Most of the suggestions of v2 [1] have been applied in this v3 revision, except the local menuconfig and mrproper targets, as explained in [2].

A fresh run with tinyconfig for ppc, ppc64 and ppc64le:

    $ for arch in ppc ppc64 ppc64le; do \
        mkdir -p $PWD/kernel-$arch;
        time make defconfig run DEFCONFIG=tinyconfig ARCH=$arch O=$PWD/kernel-$arch RUN_OUT=$PWD/run.$arch.out;
      done

rerun for ppc, ppc64 and ppc64le:

    $ for arch in ppc ppc64 ppc64le; do \
        make rerun ARCH=$arch O=$PWD/kernel-$arch RUN_OUT=$PWD/run.$arch.out;
      done
    Running /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/kernel-ppc/vmlinux on qemu-system-ppc
Quoted 1 line
Quoted 1 line

[ppc] Kernel command line: console=ttyS0 panic=-1

printk: console [ttyS0] enabled
Run /init as init process
Running test 'startup'
Running test 'syscall'
Running test 'stdlib'
Running test 'vfprintf'
Running test 'protection'
Leaving init with final status: 0
reboot: Power down
powered off, test finish
qemu-system-ppc: terminating on signal 15 from pid 190248 ()

165 test(s): 156 passed, 9 skipped, 0 failed => status: warning

See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.ppc.out
Running /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/kernel-ppc64/vmlinux on qemu-system-ppc64
Linux version 6.4.0+ (ubuntu@linux-lab) (powerpc64le-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #2 SMP Fri Jul 28 01:40:55 CST 2023 Kernel command line: console=hvc0 panic=-1
printk: console [hvc0] enabled
printk: console [hvc0] enabled
Run /init as init process
Running test 'startup'
Running test 'syscall'
Running test 'stdlib'
Running test 'vfprintf'
Running test 'protection'
Leaving init with final status: 0
reboot: Power down
powered off, test finish

165 test(s): 156 passed, 9 skipped, 0 failed => status: warning

See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.ppc64.out
Running /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/kernel-ppc64le/arch/powerpc/boot/zImage on qemu-system-ppc64le
Linux version 6.4.0+ (ubuntu@linux-lab) (powerpc64le-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #2 SMP Fri Jul 28 01:41:12 CST 2023 Kernel command line: console=hvc0 panic=-1
Run /init as init process
Running test 'startup'
Running test 'syscall'
Running test 'stdlib'
Running test 'vfprintf'
Running test 'protection'
Leaving init with final status: 0
reboot: Power down
powered off, test finish

165 test(s): 156 passed, 9 skipped, 0 failed => status: warning

See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.ppc64le.out

A fast report on existing test logs:

    $ for arch in ppc ppc64 ppc64le; do \
        make report ARCH=$arch RUN_OUT=$PWD/run.$arch.out | grep status; \
      done
    165 test(s): 156 passed,   9 skipped,   0 failed => status: warning
    165 test(s): 156 passed,   9 skipped,   0 failed => status: warning
    165 test(s): 156 passed,   9 skipped,   0 failed => status: warning

Changes from v2 --> v3:

* selftests/nolibc: allow report with existing test log selftests/nolibc: fix up O= option support selftests/nolibc: allow customize CROSS_COMPILE by architecture selftests/nolibc: customize CROSS_COMPILE for 32/64-bit powerpc selftests/nolibc: tinyconfig: add extra common options

No Change.

* selftests/nolibc: add macros to reduce duplicated changes

Remove REPORT_RUN_OUT and LOG_OUT.

* selftests/nolibc: string the core targets

Removed extconfig target from our v3 powerpc patchset [3], the operations have been merged into the defconfig target.

Let kernel depends on $(KERNEL_CONFIG) instead of the removed extconfig.

* selftests/nolibc: add menuconfig and mrproper for development

like the other local nolibc targets, still require local menuconfig and mrproper targets for consistent usage with the same ARCH and no -C /path/to/srctree

Merge them together to reduce duplicated entries.

* selftests/nolibc: allow quit qemu-system when poweroff fails

Enhance timeout logic with more expected strings print and detection about the booting of bios, kernel, init and test.

Add a default 10 seconds of QEMU_TIMEOUT for every architecture to detect all of the potential boog hang or failed poweroff.

* selftests/nolibc: customize QEMU_TIMEOUT for ppc64/ppc64le

Reduce QEMU_TIMEOUT from 60 seconds to a more normal 15 and 20 seconds for ppc64 and ppc64le respectively. the main time cost is the slow bios used.

* selftests/nolibc: tinyconfig: add support for 32/64-bit powerpc

Rename the file names to shorter ones as suggestions from the powerpc patchset.

* selftests/nolibc: speed up some targets with multiple jobs

New to speed up with -j<N> by default.

Best regards,
Zhangjin
---
[1]: https://lore.kernel.org/lkml/[email protected]/
[2]: https://lore.kernel.org/lkml/[email protected]/
[3]: https://lore.kernel.org/lkml/8e9e5ac6283c6ec2ecf10a70ce55b219028497c1.1690468707.git.falcon@tinylab.org/

Zhangjin Wu (12):
selftests/nolibc: allow report with existing test log selftests/nolibc: add macros to reduce duplicated changes selftests/nolibc: fix up O= option support selftests/nolibc: string the core targets selftests/nolibc: allow customize CROSS_COMPILE by architecture selftests/nolibc: customize CROSS_COMPILE for 32/64-bit powerpc selftests/nolibc: add menuconfig and mrproper for development selftests/nolibc: allow quit qemu-system when poweroff fails selftests/nolibc: customize QEMU_TIMEOUT for ppc64/ppc64le selftests/nolibc: tinyconfig: add extra common options selftests/nolibc: tinyconfig: add support for 32/64-bit powerpc selftests/nolibc: speed up some targets with multiple jobs

 tools/testing/selftests/nolibc/Makefile       | 102 ++++++++++++++----
 .../selftests/nolibc/configs/common.config    |   4 +
 .../selftests/nolibc/configs/ppc.config       |   3 +
 .../selftests/nolibc/configs/ppc64.config     |   3 +
 .../selftests/nolibc/configs/ppc64le.config   |   4 +
 5 files changed, 98 insertions(+), 18 deletions(-)
 create mode 100644 tools/testing/selftests/nolibc/configs/common.config
 create mode 100644 tools/testing/selftests/nolibc/configs/ppc64.config
 create mode 100644 tools/testing/selftests/nolibc/configs/ppc64le.config
Signature · 1 line
2.25.1
#1 lore raw

PATCH v3 1/12 selftests/nolibc: allow report with existing test log

After the tests finish, it is valuable to report and summarize with existing test log.

This avoid rerun or run the tests again when not necessary.

---
 tools/testing/selftests/nolibc/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/nolibc/Makefile +4 -0
index 2e9694370913..75419b695f0d 100644
@@ -207,6 +207,10 @@ rerun:
 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
 	$(Q)$(REPORT) $(CURDIR)/run.out
 
+# report with existing test log
+report:
+	$(Q)$(REPORT_RUN_OUT)
+
 clean:
 	$(call QUIET_CLEAN, sysroot)
 	$(Q)rm -rf sysroot
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH v3 01/12] selftests/nolibc: allow report with existing test log

Quoted 15 lines · Zhangjin Wu

After the tests finish, it is valuable to report and summarize with existing test log.

This avoid rerun or run the tests again when not necessary.

---
 tools/testing/selftests/nolibc/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/nolibc/Makefile +0 -0
index 2e9694370913..75419b695f0d 100644

tools/testing/selftests/nolibc/Makefile rerun: · line 212

Quoted 6 lines
tools/testing/selftests/nolibc/Makefile +3 -0
@@ -207,6 +207,10 @@ rerun:
 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
 	$(Q)$(REPORT) $(CURDIR)/run.out
 
+# report with existing test log
+report:
+	$(Q)$(REPORT_RUN_OUT)

Isn't REPORT_RUN_OUT gone in this revision?

Quoted 7 lines
tools/testing/selftests/nolibc/Makefile +1 -0
+
 clean:
 	$(call QUIET_CLEAN, sysroot)
 	$(Q)rm -rf sysroot
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH v3 01/12] selftests/nolibc: allow report with existing test log

Quoted 19 lines · Zhangjin Wu
Quoted 18 lines

After the tests finish, it is valuable to report and summarize with existing test log.

This avoid rerun or run the tests again when not necessary.

---
 tools/testing/selftests/nolibc/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/nolibc/Makefile +0 -0
index 2e9694370913..75419b695f0d 100644
@@ -207,6 +207,10 @@ rerun:
 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
 	$(Q)$(REPORT) $(CURDIR)/run.out
 
Quoted 6 lines

tools/testing/selftests/nolibc/Makefile rerun: · line 212

Quoted 3 lines
tools/testing/selftests/nolibc/Makefile +3 -0
+# report with existing test log
+report:
+	$(Q)$(REPORT_RUN_OUT)

Isn't REPORT_RUN_OUT gone in this revision?

Yeah, I moved it as the first generic patch but forgot it have used a later macro, and therefore no recheck in this revision, thanks a lot.

Thanks
Zhangjin

Quoted 7 lines
Quoted 7 lines
tools/testing/selftests/nolibc/Makefile +1 -0
+
 clean:
 	$(call QUIET_CLEAN, sysroot)
 	$(Q)rm -rf sysroot
Signature · 1 line
2.25.1
#4 lore raw

PATCH v3 2/12 selftests/nolibc: add macros to reduce duplicated changes

In reply to Zhangjin Wu

The kernel targets share the same kernel make operations, the same .config file, the same kernel image, add MAKE_KERNEL, KERNEL_CONFIG and KERNEL_IMAGE for them.

Many targets use the same log file, add RUN_OUT to allow save log by architecture, for example: 'make RUN_OUT=$PWD/run.$arch.out'.

The qemu run/rerun targets share the same qemu system run command, add QEMU_SYSTEM_RUN for them.

---
 tools/testing/selftests/nolibc/Makefile | 41 ++++++++++++++++---------
 1 file changed, 26 insertions(+), 15 deletions(-)
tools/testing/selftests/nolibc/Makefile +26 -15
index 75419b695f0d..bfea1ea0b4e7 100644
@@ -169,47 +169,58 @@ endif
 libc-test: nolibc-test.c
 	$(QUIET_CC)$(CC) -o $@ $<
 
+# common macros for logging
+RUN_OUT = $(CURDIR)/run.out
+
 # local libc-test
 run-libc-test: libc-test
-	$(Q)./libc-test > "$(CURDIR)/run.out" || :
-	$(Q)$(REPORT) $(CURDIR)/run.out
+	$(Q)./libc-test > "$(RUN_OUT)" || :
+	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # local nolibc-test
 run-nolibc-test: nolibc-test
-	$(Q)./nolibc-test > "$(CURDIR)/run.out" || :
-	$(Q)$(REPORT) $(CURDIR)/run.out
+	$(Q)./nolibc-test > "$(RUN_OUT)" || :
+	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # qemu user-land test
 run-user: nolibc-test
-	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
-	$(Q)$(REPORT) $(CURDIR)/run.out
+	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(RUN_OUT)" || :
+	$(Q)$(REPORT) "$(RUN_OUT)"
 
 initramfs: nolibc-test
 	$(QUIET_MKDIR)mkdir -p initramfs
 	$(call QUIET_INSTALL, initramfs/init)
 	$(Q)cp nolibc-test initramfs/init
 
+# common macros for kernel targets
+MAKE_KERNEL   = $(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
+KERNEL_CONFIG = $(srctree)/.config
+KERNEL_IMAGE  = $(srctree)/$(IMAGE)
+
 defconfig:
-	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
-	$(Q)$(srctree)/scripts/kconfig/merge_config.sh -O "$(srctree)" -m "$(srctree)/.config" $(foreach c,$(EXTCONFIG),$(wildcard $(CURDIR)/configs/$c))
-	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) KCONFIG_ALLCONFIG="$(srctree)/.config" allnoconfig
+	$(Q)$(MAKE_KERNEL) mrproper $(DEFCONFIG) prepare
+	$(Q)$(srctree)/scripts/kconfig/merge_config.sh -O "$(srctree)" -m "$(KERNEL_CONFIG)" $(foreach c,$(EXTCONFIG),$(wildcard $(CURDIR)/configs/$c))
+	$(Q)$(MAKE_KERNEL) KCONFIG_ALLCONFIG="$(KERNEL_CONFIG)" allnoconfig
 
 kernel: initramfs
-	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
+	$(Q)$(MAKE_KERNEL) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
+
+# common macros for qemu run/rerun targets
+QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS)
 
 # run the tests after building the kernel
 run: kernel
-	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
-	$(Q)$(REPORT) $(CURDIR)/run.out
+	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # re-run the tests from an existing kernel
 rerun:
-	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
-	$(Q)$(REPORT) $(CURDIR)/run.out
+	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # report with existing test log
 report:
-	$(Q)$(REPORT_RUN_OUT)
+	$(Q)$(REPORT) "$(RUN_OUT)"
 
 clean:
 	$(call QUIET_CLEAN, sysroot)
Signature · 1 line
2.25.1
#5 lore raw

PATCH v3 3/12 selftests/nolibc: fix up O= option support

In reply to Zhangjin Wu

To avoid pollute the source code tree and avoid mrproper for every architecture switch, the O= argument must be supported.

Both IMAGE and .config are from the building directory, let's use objtree instead of srctree for them.

If no O= option specified, means building kernel in source code tree, objtree should be srctree in such case.

---
 tools/testing/selftests/nolibc/Makefile | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
tools/testing/selftests/nolibc/Makefile +6 -3
index bfea1ea0b4e7..f5680b9ed85c 100644
@@ -9,6 +9,9 @@ ifeq ($(srctree),)
 srctree := $(patsubst %/tools/testing/selftests/,%,$(dir $(CURDIR)))
 endif
 
+# add objtree for O= argument, required by IMAGE and .config
+objtree ?= $(srctree)
+
 ifeq ($(ARCH),)
 include $(srctree)/scripts/subarch.include
 ARCH = $(SUBARCH)
@@ -194,12 +197,12 @@ initramfs: nolibc-test
 
 # common macros for kernel targets
 MAKE_KERNEL   = $(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
-KERNEL_CONFIG = $(srctree)/.config
-KERNEL_IMAGE  = $(srctree)/$(IMAGE)
+KERNEL_CONFIG = $(objtree)/.config
+KERNEL_IMAGE  = $(objtree)/$(IMAGE)
 
 defconfig:
 	$(Q)$(MAKE_KERNEL) mrproper $(DEFCONFIG) prepare
-	$(Q)$(srctree)/scripts/kconfig/merge_config.sh -O "$(srctree)" -m "$(KERNEL_CONFIG)" $(foreach c,$(EXTCONFIG),$(wildcard $(CURDIR)/configs/$c))
+	$(Q)$(srctree)/scripts/kconfig/merge_config.sh -O "$(objtree)" -m "$(KERNEL_CONFIG)" $(foreach c,$(EXTCONFIG),$(wildcard $(CURDIR)/configs/$c))
 	$(Q)$(MAKE_KERNEL) KCONFIG_ALLCONFIG="$(KERNEL_CONFIG)" allnoconfig
 
 kernel: initramfs
Signature · 1 line
2.25.1
#6 lore raw

PATCH v3 4/12 selftests/nolibc: string the core targets

In reply to Zhangjin Wu

To avoid run targets one by one manually and boringly, let's string them with IMAGE and .config, the MAKE command will trigger the dependencies for us.

Note, defconfig target is only triggered while the .config is not there, it means only trigger defconfig for the first run or after a mrproper.

---
 tools/testing/selftests/nolibc/Makefile | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
tools/testing/selftests/nolibc/Makefile +11 -3
index f5680b9ed85c..3a61fa7e42a0 100644
@@ -153,6 +153,7 @@ all: run
 
 sysroot: sysroot/$(ARCH)/include
 
+PHONY = sysroot/$(ARCH)/include
 sysroot/$(ARCH)/include:
 	$(Q)rm -rf sysroot/$(ARCH) sysroot/sysroot
 	$(QUIET_MKDIR)mkdir -p sysroot
@@ -205,14 +206,21 @@ defconfig:
 	$(Q)$(srctree)/scripts/kconfig/merge_config.sh -O "$(objtree)" -m "$(KERNEL_CONFIG)" $(foreach c,$(EXTCONFIG),$(wildcard $(CURDIR)/configs/$c))
 	$(Q)$(MAKE_KERNEL) KCONFIG_ALLCONFIG="$(KERNEL_CONFIG)" allnoconfig
 
-kernel: initramfs
+PHONY += $(KERNEL_CONFIG)
+$(KERNEL_CONFIG):
+	$(Q)if [ ! -f "$(KERNEL_CONFIG)" ]; then $(MAKE) --no-print-directory defconfig; fi
+
+kernel: $(KERNEL_CONFIG)
+	$(Q)$(MAKE) --no-print-directory initramfs
 	$(Q)$(MAKE_KERNEL) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
 
 # common macros for qemu run/rerun targets
 QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS)
 
 # run the tests after building the kernel
-run: kernel
+PHONY += $(KERNEL_IMAGE)
+$(KERNEL_IMAGE): kernel
+run: $(KERNEL_IMAGE)
 	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
 	$(Q)$(REPORT) "$(RUN_OUT)"
 
@@ -237,4 +245,4 @@ clean:
 	$(call QUIET_CLEAN, run.out)
 	$(Q)rm -rf run.out
 
-.PHONY: sysroot/$(ARCH)/include
+.PHONY: $(PHONY)
Signature · 1 line
2.25.1
#7 lore raw

PATCH v3 5/12 selftests/nolibc: allow customize CROSS_COMPILE by architecture

In reply to Zhangjin Wu

Some cross compilers may not just be prefixed with ARCH, customize them by architecture may simplify the test a lot, especially, when iterate with ARCH.

After customizing this for every architecture, the minimal test argument will be architecture itself, no CROSS_COMPILE required to be passed.

If the prefix of installed cross compiler is not the same as the one customized, we can also pass CROSS_COMPILE as before or even pass CROSS_COMPILE_<ARCH>.

---
 tools/testing/selftests/nolibc/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
tools/testing/selftests/nolibc/Makefile +6 -0
index 3a61fa7e42a0..3f15c7f7ef76 100644
@@ -45,6 +45,12 @@ IMAGE_loongarch  = arch/loongarch/boot/vmlinuz.efi
 IMAGE            = $(IMAGE_$(XARCH))
 IMAGE_NAME       = $(notdir $(IMAGE))
 
+# CROSS_COMPILE: cross toolchain prefix by architecture
+CROSS_COMPILE           ?= $(CROSS_COMPILE_$(XARCH))
+
+# make sure CC is prefixed with CROSS_COMPILE
+$(call allow-override,CC,$(CROSS_COMPILE)gcc)
+
 # default kernel configurations that appear to be usable
 DEFCONFIG_i386       = defconfig
 DEFCONFIG_x86_64     = defconfig
Signature · 1 line
2.25.1
#8 lore raw

PATCH v3 6/12 selftests/nolibc: customize CROSS_COMPILE for 32/64-bit powerpc

In reply to Zhangjin Wu

The little-endian powerpc64le compilers provided by Ubuntu and Fedora are able to compile big endian kernel and big endian nolibc-test [1].

These default CROSS_COMPILE settings allow to test target architectures with:

    $ cd /path/to/tools/testing/selftests/nolibc/
    $ for arch in ppc ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

If want to use another cross compiler, please simply pass CROSS_COMPILE or CC as before.

For example, it is able to build 64-bit nolibc-test with the big endian powerpc64-linux-gcc crosstool from [2]:

    $ wget -c https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.1.0/x86_64-gcc-13.1.0-nolibc-powerpc64-linux.tar.xz
    $ tar xvf x86_64-gcc-13.1.0-nolibc-powerpc64-linux.tar.xz
    $ export PATH=$PWD/gcc-13.1.0-nolibc/powerpc64-linux/bin/:$PATH
    $ export CROSS_COMPILE_ppc64=powerpc64-linux-
    $ export CROSS_COMPILE_ppc64le=powerpc64-linux-
    $ for arch in ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

Or specify CC directly with full path:

    $ export CC=$PWD/gcc-13.1.0-nolibc/powerpc64-linux/bin/powerpc64-linux-gcc
    $ for arch in ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

[1]: https://github.com/open-power/skiboot
[2]: https://mirrors.edge.kernel.org/pub/tools/crosstool/

---
 tools/testing/selftests/nolibc/Makefile | 3 +++
 1 file changed, 3 insertions(+)
tools/testing/selftests/nolibc/Makefile +3 -0
index 3f15c7f7ef76..6385915d16c9 100644
@@ -46,6 +46,9 @@ IMAGE            = $(IMAGE_$(XARCH))
 IMAGE_NAME       = $(notdir $(IMAGE))
 
 # CROSS_COMPILE: cross toolchain prefix by architecture
+CROSS_COMPILE_ppc       ?= powerpc-linux-gnu-
+CROSS_COMPILE_ppc64     ?= powerpc64le-linux-gnu-
+CROSS_COMPILE_ppc64le   ?= powerpc64le-linux-gnu-
 CROSS_COMPILE           ?= $(CROSS_COMPILE_$(XARCH))
 
 # make sure CC is prefixed with CROSS_COMPILE
Signature · 1 line
2.25.1
maintainer
#9 lore raw

Re: [PATCH v3 06/12] selftests/nolibc: customize CROSS_COMPILE for 32/64-bit powerpc

Quoted 48 lines · Zhangjin Wu

The little-endian powerpc64le compilers provided by Ubuntu and Fedora are able to compile big endian kernel and big endian nolibc-test [1].

These default CROSS_COMPILE settings allow to test target architectures with:

    $ cd /path/to/tools/testing/selftests/nolibc/
    $ for arch in ppc ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

If want to use another cross compiler, please simply pass CROSS_COMPILE or CC as before.

For example, it is able to build 64-bit nolibc-test with the big endian powerpc64-linux-gcc crosstool from [2]:

    $ wget -c https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.1.0/x86_64-gcc-13.1.0-nolibc-powerpc64-linux.tar.xz
    $ tar xvf x86_64-gcc-13.1.0-nolibc-powerpc64-linux.tar.xz
    $ export PATH=$PWD/gcc-13.1.0-nolibc/powerpc64-linux/bin/:$PATH
    $ export CROSS_COMPILE_ppc64=powerpc64-linux-
    $ export CROSS_COMPILE_ppc64le=powerpc64-linux-
    $ for arch in ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

Or specify CC directly with full path:

    $ export CC=$PWD/gcc-13.1.0-nolibc/powerpc64-linux/bin/powerpc64-linux-gcc
    $ for arch in ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

[1]: https://github.com/open-power/skiboot
[2]: https://mirrors.edge.kernel.org/pub/tools/crosstool/

---
 tools/testing/selftests/nolibc/Makefile | 3 +++
 1 file changed, 3 insertions(+)
tools/testing/selftests/nolibc/Makefile +0 -0
index 3f15c7f7ef76..6385915d16c9 100644

tools/testing/selftests/nolibc/Makefile IMAGE = $(IMAGE_$(XARCH)) · line 52

Quoted 7 lines
tools/testing/selftests/nolibc/Makefile +3 -0
@@ -46,6 +46,9 @@ IMAGE            = $(IMAGE_$(XARCH))
 IMAGE_NAME       = $(notdir $(IMAGE))
 
 # CROSS_COMPILE: cross toolchain prefix by architecture
+CROSS_COMPILE_ppc       ?= powerpc-linux-gnu-
+CROSS_COMPILE_ppc64     ?= powerpc64le-linux-gnu-
+CROSS_COMPILE_ppc64le   ?= powerpc64le-linux-gnu-
 CROSS_COMPILE           ?= $(CROSS_COMPILE_$(XARCH))

It seems to me that this patch and the previous one were rather for the PPC series as I'm not seeing the relation with the tiny config here.

Willy

#10 lore raw

Re: [PATCH v3 06/12] selftests/nolibc: customize CROSS_COMPILE for 32/64-bit powerpc

Quoted 51 lines · Zhangjin Wu
Quoted 50 lines

The little-endian powerpc64le compilers provided by Ubuntu and Fedora are able to compile big endian kernel and big endian nolibc-test [1].

These default CROSS_COMPILE settings allow to test target architectures with:

    $ cd /path/to/tools/testing/selftests/nolibc/
    $ for arch in ppc ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

If want to use another cross compiler, please simply pass CROSS_COMPILE or CC as before.

For example, it is able to build 64-bit nolibc-test with the big endian powerpc64-linux-gcc crosstool from [2]:

    $ wget -c https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.1.0/x86_64-gcc-13.1.0-nolibc-powerpc64-linux.tar.xz
    $ tar xvf x86_64-gcc-13.1.0-nolibc-powerpc64-linux.tar.xz
    $ export PATH=$PWD/gcc-13.1.0-nolibc/powerpc64-linux/bin/:$PATH
    $ export CROSS_COMPILE_ppc64=powerpc64-linux-
    $ export CROSS_COMPILE_ppc64le=powerpc64-linux-
    $ for arch in ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

Or specify CC directly with full path:

    $ export CC=$PWD/gcc-13.1.0-nolibc/powerpc64-linux/bin/powerpc64-linux-gcc
    $ for arch in ppc64 ppc64le; do \
        make run-user ARCH=$arch | grep "status: "; \
      done

[1]: https://github.com/open-power/skiboot
[2]: https://mirrors.edge.kernel.org/pub/tools/crosstool/

---
 tools/testing/selftests/nolibc/Makefile | 3 +++
 1 file changed, 3 insertions(+)
tools/testing/selftests/nolibc/Makefile +0 -0
index 3f15c7f7ef76..6385915d16c9 100644
@@ -46,6 +46,9 @@ IMAGE            = $(IMAGE_$(XARCH))
 IMAGE_NAME       = $(notdir $(IMAGE))
 
Quoted 10 lines

tools/testing/selftests/nolibc/Makefile IMAGE = $(IMAGE_$(XARCH)) · line 52

Quoted 5 lines
tools/testing/selftests/nolibc/Makefile +3 -0
 # CROSS_COMPILE: cross toolchain prefix by architecture
+CROSS_COMPILE_ppc       ?= powerpc-linux-gnu-
+CROSS_COMPILE_ppc64     ?= powerpc64le-linux-gnu-
+CROSS_COMPILE_ppc64le   ?= powerpc64le-linux-gnu-
 CROSS_COMPILE           ?= $(CROSS_COMPILE_$(XARCH))

It seems to me that this patch and the previous one were rather for the PPC series as I'm not seeing the relation with the tiny config here.

Yes, it is also ok for the powerpc series, they mainly aim to the fast build and test goal of 'tinyconfig', and the other default CROSS_COMPILE's will be added together with the left tinyconfig support by architecture.

I'm ok if you are happy to merge it into the powerpc series, then, we can focus on the left ones ;-)

Thanks,
Zhangjin

Quoted 1 line

Willy

#11 lore raw

PATCH v3 7/12 selftests/nolibc: add menuconfig and mrproper for development

In reply to Zhangjin Wu

menuconfig and mrproper are frequently used operations during a new architecture porting, testing or debugging.

menuconfig is required to tune and test extra kernel config options for tinyconfig.

mrproper is required to get a clean srctree while want to start a new building with O= option, the old generated files in srctree must be mrproper-ed.

differ from local nolibc targets, the menuconfig and mrproper targets from top-level Makefile accept different ARCH variable and require extra '-C /path/to/srctree', which make development not consistent and therefore very painful, let's add local menuconfig and mrproper targets too.

To reduce duplicated entries, menuconfig and mrproper are added together.

---
 tools/testing/selftests/nolibc/Makefile | 3 +++
 1 file changed, 3 insertions(+)
tools/testing/selftests/nolibc/Makefile +3 -0
index 6385915d16c9..a214745e0f3e 100644
@@ -215,6 +215,9 @@ defconfig:
 	$(Q)$(srctree)/scripts/kconfig/merge_config.sh -O "$(objtree)" -m "$(KERNEL_CONFIG)" $(foreach c,$(EXTCONFIG),$(wildcard $(CURDIR)/configs/$c))
 	$(Q)$(MAKE_KERNEL) KCONFIG_ALLCONFIG="$(KERNEL_CONFIG)" allnoconfig
 
+menuconfig mrproper:
+	$(Q)$(MAKE_KERNEL) $@
+
 PHONY += $(KERNEL_CONFIG)
 $(KERNEL_CONFIG):
 	$(Q)if [ ! -f "$(KERNEL_CONFIG)" ]; then $(MAKE) --no-print-directory defconfig; fi
Signature · 1 line
2.25.1
#12 lore raw

PATCH v3 8/12 selftests/nolibc: allow quit qemu-system when poweroff fails

In reply to Zhangjin Wu

The kernel of some architectures can not poweroff qemu-system normally, especially for tinyconfig.

Some architectures may have no kernel poweroff support, the others may require more kernel config options and therefore slow down the tinyconfig build and test. and also, it's very hard (and some even not possible) to find out the exact poweroff related kernel config options for every architecture.

Since the low-level poweroff support is heavily kernel & qemu dependent, it is not that critical to both nolibc and nolibc-test, let's simply ignore the poweroff required kernel config options for tinyconfig (and even for defconfig) and quit qemu-system after a specified timeout or with an expected system halt or poweroff string (these strings mean our reboot() library routine is perfectly ok).

QEMU_TIMEOUT can be configured for every architecture based on their time cost requirement of bios boot + kernel boot + test + poweroff.

By default, 10 seconds timeout is configured, this is enough for most of the architectures, otherwise, customize one by architecture.

To tell users the test running progress in time, some critical running status are also printed and detected.

---
 tools/testing/selftests/nolibc/Makefile | 30 +++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
tools/testing/selftests/nolibc/Makefile +28 -2
index a214745e0f3e..9a57de3b283c 100644
@@ -105,6 +105,9 @@ QEMU_ARGS_s390       = -M s390-ccw-virtio -m 1G -append "console=ttyS0 panic=-1
 QEMU_ARGS_loongarch  = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
 QEMU_ARGS            = $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_EXTRA)
 
+# QEMU_TIMEOUT: some architectures can not poweroff normally, especially for tinyconfig
+QEMU_TIMEOUT           = $(or $(QEMU_TIMEOUT_$(XARCH)),10)
+
 # OUTPUT is only set when run from the main makefile, otherwise
 # it defaults to this nolibc directory.
 OUTPUT ?= $(CURDIR)/
@@ -229,16 +232,39 @@ kernel: $(KERNEL_CONFIG)
 # common macros for qemu run/rerun targets
 QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS)
 
+TIMEOUT_CMD = t=$(QEMU_TIMEOUT); past=0; \
+	bios_timeout=$$(expr $$t - 7); kernel_timeout=$$(expr $$t - 5); init_timeout=$$(expr $$t - 3); test_timeout=$$(expr $$t - 1);              \
+	err=""; bios=0; kernel=0; init=0; test=0; poweredoff=0; panic=0;                                                                           \
+	echo "Running $(KERNEL_IMAGE) on qemu-system-$(QEMU_ARCH)";                                                                                \
+	while [ $$t -gt 0 ]; do                                                                                                                    \
+	    sleep 2; t=$$(expr $$t - 2); past=$$(expr $$past + 2);                                                                                 \
+	    if [ $$bios -eq 0 ] && grep -E "Linux version|Kernel command line|printk: console" "$(RUN_OUT)"; then bios=1; fi;                      \
+	    if [ $$bios -eq 1 -a $$kernel -eq 0 ] && grep -E "Run .* as init process" "$(RUN_OUT)"; then kernel=1; fi;                             \
+	    if [ $$kernel -eq 1 -a $$init -eq 0 ] && grep -E "Running test" "$(RUN_OUT)"; then init=1; fi;                                         \
+	    if [ $$init -eq 1 -a $$test -eq 0 ] && grep -E "Leaving init with final status|Exiting with status" "$(RUN_OUT)"; then test=1; fi;     \
+	    if [ $$init -eq 1 ] && grep -E "Kernel panic - not syncing: Attempted to kill init" "$(RUN_OUT)"; then err="test"; sleep 1; break; fi; \
+	    if [ $$test -eq 1 ] && grep -E "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then poweredoff=1; sleep 1; break; fi;         \
+	    if [ $$past -gt $$bios_timeout -a $$bios -eq 0 ]; then err="bios"; break; fi;                                                          \
+	    if [ $$past -gt $$kernel_timeout -a $$kernel -eq 0 ]; then err="kernel"; break; fi;                                                    \
+	    if [ $$past -gt $$init_timeout -a $$init -eq 0 ]; then err="init"; break; fi;                                                          \
+	    if [ $$past -gt $$test_timeout -a $$test -eq 0 ]; then err="test"; break; fi;                                                          \
+	done;                                                                                                                                      \
+	if [ -z "$$err" -a $$poweredoff -eq 0 -a $$panic -eq 0 ]; then err="qemu-system-$(QEMU_ARCH)"; fi;                                         \
+	if [ -n "$$err" ]; then echo "$$err may timeout, test failed"; tail -10 $(RUN_OUT); else echo "powered off, test finish"; fi;              \
+	pkill -15 qemu-system-$(QEMU_ARCH) || true
+
+TIMEOUT_QEMU_RUN = ($(QEMU_SYSTEM_RUN) > "$(RUN_OUT)" &); $(TIMEOUT_CMD)
+
 # run the tests after building the kernel
 PHONY += $(KERNEL_IMAGE)
 $(KERNEL_IMAGE): kernel
 run: $(KERNEL_IMAGE)
-	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(TIMEOUT_QEMU_RUN)
 	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # re-run the tests from an existing kernel
 rerun:
-	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(TIMEOUT_QEMU_RUN)
 	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # report with existing test log
Signature · 1 line
2.25.1
#13 lore raw

PATCH v3 8/12 selftests/nolibc: allow quit qemu-system when poweroff fails

Hi, Willy

two trivial updates required in this patch.

[...]

Quoted 4 lines

To tell users the test running progress in time, some critical running status are also printed and detected.

[...]

tools/testing/selftests/nolibc/Makefile kernel: $(KERNEL_CONFIG) · line 237

Quoted 7 lines
tools/testing/selftests/nolibc/Makefile +3 -0
@@ -229,16 +232,39 @@ kernel: $(KERNEL_CONFIG)
 # common macros for qemu run/rerun targets
 QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS)
 
+TIMEOUT_CMD = t=$(QEMU_TIMEOUT); past=0; \
+	bios_timeout=$$(expr $$t - 7); kernel_timeout=$$(expr $$t - 5); init_timeout=$$(expr $$t - 3); test_timeout=$$(expr $$t - 1);              \
+	err=""; bios=0; kernel=0; init=0; test=0; poweredoff=0; panic=0;                                                                           \

This 'panic=0;' variable init should be removed, it is not required in the latest version:

	err=""; bios=0; kernel=0; init=0; test=0; poweredoff=0;                                                                                    \

tools/testing/selftests/nolibc/Makefile kernel: $(KERNEL_CONFIG) · line 244

Quoted 7 lines
tools/testing/selftests/nolibc/Makefile +7 -0
+	echo "Running $(KERNEL_IMAGE) on qemu-system-$(QEMU_ARCH)";                                                                                \
+	while [ $$t -gt 0 ]; do                                                                                                                    \
+	    sleep 2; t=$$(expr $$t - 2); past=$$(expr $$past + 2);                                                                                 \
+	    if [ $$bios -eq 0 ] && grep -E "Linux version|Kernel command line|printk: console" "$(RUN_OUT)"; then bios=1; fi;                      \
+	    if [ $$bios -eq 1 -a $$kernel -eq 0 ] && grep -E "Run .* as init process" "$(RUN_OUT)"; then kernel=1; fi;                             \
+	    if [ $$kernel -eq 1 -a $$init -eq 0 ] && grep -E "Running test" "$(RUN_OUT)"; then init=1; fi;                                         \
+	    if [ $$init -eq 1 -a $$test -eq 0 ] && grep -E "Leaving init with final status|Exiting with status" "$(RUN_OUT)"; then test=1; fi;     \

It is better to get the line of 'Total number of errors' instead of 'Exiting with status', the later never trigger in qemu-system run.

	    if [ $$init -eq 1 -a $$test -eq 0 ] && grep -E "Leaving init with final status|Total number of errors" "$(RUN_OUT)"; then test=1; fi;  \

tools/testing/selftests/nolibc/Makefile kernel: $(KERNEL_CONFIG) · line 252

Quoted 8 lines
tools/testing/selftests/nolibc/Makefile +8 -0
+	    if [ $$init -eq 1 ] && grep -E "Kernel panic - not syncing: Attempted to kill init" "$(RUN_OUT)"; then err="test"; sleep 1; break; fi; \
+	    if [ $$test -eq 1 ] && grep -E "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then poweredoff=1; sleep 1; break; fi;         \
+	    if [ $$past -gt $$bios_timeout -a $$bios -eq 0 ]; then err="bios"; break; fi;                                                          \
+	    if [ $$past -gt $$kernel_timeout -a $$kernel -eq 0 ]; then err="kernel"; break; fi;                                                    \
+	    if [ $$past -gt $$init_timeout -a $$init -eq 0 ]; then err="init"; break; fi;                                                          \
+	    if [ $$past -gt $$test_timeout -a $$test -eq 0 ]; then err="test"; break; fi;                                                          \
+	done;                                                                                                                                      \
+	if [ -z "$$err" -a $$poweredoff -eq 0 -a $$panic -eq 0 ]; then err="qemu-system-$(QEMU_ARCH)"; fi;                                         \

And here, we should remove the panic check here too, it is replaced with 'err="test"':

	if [ -z "$$err" -a $$poweredoff -eq 0 ]; then err="qemu-system-$(QEMU_ARCH)"; fi;                                                          \

Thanks,
Zhangjin

Quoted 22 lines
tools/testing/selftests/nolibc/Makefile +7 -2
+	if [ -n "$$err" ]; then echo "$$err may timeout, test failed"; tail -10 $(RUN_OUT); else echo "powered off, test finish"; fi;              \
+	pkill -15 qemu-system-$(QEMU_ARCH) || true
+
+TIMEOUT_QEMU_RUN = ($(QEMU_SYSTEM_RUN) > "$(RUN_OUT)" &); $(TIMEOUT_CMD)
+
 # run the tests after building the kernel
 PHONY += $(KERNEL_IMAGE)
 $(KERNEL_IMAGE): kernel
 run: $(KERNEL_IMAGE)
-	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(TIMEOUT_QEMU_RUN)
 	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # re-run the tests from an existing kernel
 rerun:
-	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(TIMEOUT_QEMU_RUN)
 	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # report with existing test log
Signature · 1 line
2.25.1
#14 lore raw

Re: [PATCH v3 08/12] selftests/nolibc: allow quit qemu-system when poweroff fails

In reply to Zhangjin Wu
Quoted 63 lines · Zhangjin Wu

The kernel of some architectures can not poweroff qemu-system normally, especially for tinyconfig.

Some architectures may have no kernel poweroff support, the others may require more kernel config options and therefore slow down the tinyconfig build and test. and also, it's very hard (and some even not possible) to find out the exact poweroff related kernel config options for every architecture.

Since the low-level poweroff support is heavily kernel & qemu dependent, it is not that critical to both nolibc and nolibc-test, let's simply ignore the poweroff required kernel config options for tinyconfig (and even for defconfig) and quit qemu-system after a specified timeout or with an expected system halt or poweroff string (these strings mean our reboot() library routine is perfectly ok).

QEMU_TIMEOUT can be configured for every architecture based on their time cost requirement of bios boot + kernel boot + test + poweroff.

By default, 10 seconds timeout is configured, this is enough for most of the architectures, otherwise, customize one by architecture.

To tell users the test running progress in time, some critical running status are also printed and detected.

---
 tools/testing/selftests/nolibc/Makefile | 30 +++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
tools/testing/selftests/nolibc/Makefile +16 -0
index a214745e0f3e..9a57de3b283c 100644
@@ -105,6 +105,9 @@ QEMU_ARGS_s390       = -M s390-ccw-virtio -m 1G -append "console=ttyS0 panic=-1
 QEMU_ARGS_loongarch  = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
 QEMU_ARGS            = $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_EXTRA)
 
+# QEMU_TIMEOUT: some architectures can not poweroff normally, especially for tinyconfig
+QEMU_TIMEOUT           = $(or $(QEMU_TIMEOUT_$(XARCH)),10)
+
 # OUTPUT is only set when run from the main makefile, otherwise
 # it defaults to this nolibc directory.
 OUTPUT ?= $(CURDIR)/
@@ -229,16 +232,39 @@ kernel: $(KERNEL_CONFIG)
 # common macros for qemu run/rerun targets
 QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS)
 
+TIMEOUT_CMD = t=$(QEMU_TIMEOUT); past=0; \
+	bios_timeout=$$(expr $$t - 7); kernel_timeout=$$(expr $$t - 5); init_timeout=$$(expr $$t - 3); test_timeout=$$(expr $$t - 1);              \
+	err=""; bios=0; kernel=0; init=0; test=0; poweredoff=0; panic=0;                                                                           \
+	echo "Running $(KERNEL_IMAGE) on qemu-system-$(QEMU_ARCH)";                                                                                \
+	while [ $$t -gt 0 ]; do                                                                                                                    \
+	    sleep 2; t=$$(expr $$t - 2); past=$$(expr $$past + 2);                                                                                 \
+	    if [ $$bios -eq 0 ] && grep -E "Linux version|Kernel command line|printk: console" "$(RUN_OUT)"; then bios=1; fi;                      \
+	    if [ $$bios -eq 1 -a $$kernel -eq 0 ] && grep -E "Run .* as init process" "$(RUN_OUT)"; then kernel=1; fi;                             \
+	    if [ $$kernel -eq 1 -a $$init -eq 0 ] && grep -E "Running test" "$(RUN_OUT)"; then init=1; fi;                                         \
+	    if [ $$init -eq 1 -a $$test -eq 0 ] && grep -E "Leaving init with final status|Exiting with status" "$(RUN_OUT)"; then test=1; fi;     \
+	    if [ $$init -eq 1 ] && grep -E "Kernel panic - not syncing: Attempted to kill init" "$(RUN_OUT)"; then err="test"; sleep 1; break; fi; \
+	    if [ $$test -eq 1 ] && grep -E "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then poweredoff=1; sleep 1; break; fi;         \
+	    if [ $$past -gt $$bios_timeout -a $$bios -eq 0 ]; then err="bios"; break; fi;                                                          \

tools/testing/selftests/nolibc/Makefile kernel: $(KERNEL_CONFIG) · line 257

Quoted 10 lines
tools/testing/selftests/nolibc/Makefile +10 -0
+	    if [ $$past -gt $$kernel_timeout -a $$kernel -eq 0 ]; then err="kernel"; break; fi;                                                    \
+	    if [ $$past -gt $$init_timeout -a $$init -eq 0 ]; then err="init"; break; fi;                                                          \
+	    if [ $$past -gt $$test_timeout -a $$test -eq 0 ]; then err="test"; break; fi;                                                          \
+	done;                                                                                                                                      \
+	if [ -z "$$err" -a $$poweredoff -eq 0 -a $$panic -eq 0 ]; then err="qemu-system-$(QEMU_ARCH)"; fi;                                         \
+	if [ -n "$$err" ]; then echo "$$err may timeout, test failed"; tail -10 $(RUN_OUT); else echo "powered off, test finish"; fi;              \
+	pkill -15 qemu-system-$(QEMU_ARCH) || true
+
+TIMEOUT_QEMU_RUN = ($(QEMU_SYSTEM_RUN) > "$(RUN_OUT)" &); $(TIMEOUT_CMD)
+

This feels fairly hacky.

Before we complicated nolibc-test to handle the no-procfs case to save a few seconds building the kernel and now we have fairly big timeouts. And a statemachine that relies on the specific strings emitted by the testsuite.

I would like to get back to something more deterministic and obvious, even at the cost of some time spent compiling the test kernels. (saying this as somebody developing on a 2016 ultrabook)

"Since the low-level poweroff support is heavily kernel & qemu dependent"

The kernel we can control.

How common are qemus with that are missing poweroff support? As this worked before I guess the only architecture where this could pose a problem would be ppc.

An alternative I would like to put up for discussion:

qemu could provide a watchdog device that is pinged by nolibc-test for each testcase.
After nolibc-test is done and didn't poweroff properly the watchdog will reset the machine. ( -watchog-action poweroff ).

The disadvantages are that we would need to add watchdog drivers to the kernels and figure out the correct watchdog devices and drivers for each arch.

It seems virtio-watchdog is not yet usable.

Quoted 18 lines
tools/testing/selftests/nolibc/Makefile +2 -2
 # run the tests after building the kernel
 PHONY += $(KERNEL_IMAGE)
 $(KERNEL_IMAGE): kernel
 run: $(KERNEL_IMAGE)
-	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(TIMEOUT_QEMU_RUN)
 	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # re-run the tests from an existing kernel
 rerun:
-	$(Q)$(QEMU_SYSTEM_RUN) > "$(RUN_OUT)"
+	$(Q)$(TIMEOUT_QEMU_RUN)
 	$(Q)$(REPORT) "$(RUN_OUT)"
 
 # report with existing test log
Signature · 1 line
2.25.1
maintainer
#15 lore raw

Re: [PATCH v3 08/12] selftests/nolibc: allow quit qemu-system when poweroff fails

Quoted 3 lines · Thomas Weißschuh
Quoted 2 lines · Zhangjin Wu

The kernel of some architectures can not poweroff qemu-system normally, especially for tinyconfig.

(...)

Quoted 1 line

This feels fairly hacky.

and totally unmaintainable in the long term. It may even fail for some users having localization.

Quoted 8 lines

Before we complicated nolibc-test to handle the no-procfs case to save a few seconds building the kernel and now we have fairly big timeouts. And a statemachine that relies on the specific strings emitted by the testsuite.

I would like to get back to something more deterministic and obvious, even at the cost of some time spent compiling the test kernels. (saying this as somebody developing on a 2016 ultrabook)

Agreed!

Quoted 11 lines

"Since the low-level poweroff support is heavily kernel & qemu dependent"

The kernel we can control.

How common are qemus with that are missing poweroff support? As this worked before I guess the only architecture where this could pose a problem would be ppc.

An alternative I would like to put up for discussion:

Quoted 7 lines

qemu could provide a watchdog device that is pinged by nolibc-test for each testcase.
After nolibc-test is done and didn't poweroff properly the watchdog will reset the machine. ( -watchog-action poweroff ).

The disadvantages are that we would need to add watchdog drivers to the kernels and figure out the correct watchdog devices and drivers for each arch.

It's an interesting idea, though at first glance it does not seem to have one for PPC.

I think I have a much simpler idea: we don't care about PPC32. I mean OK it can be supported if it happens to work, we will just not include it in default runs, because it will require Ctrl-C to finish, and so what ? nolibc has been in the kernel for 5 years or so, nobody ever cared about PPC, why should we suddenly break or complicate everything just to support a sub-arch that nobody found interesting to add till now?

Quoted 1 line

It seems virtio-watchdog is not yet usable.

Then it might become an option for the future when it eventually works.

Thanks,
Willy

#16 lore raw

Re: [PATCH v3 08/12] selftests/nolibc: allow quit qemu-system when poweroff fails

Quoted 10 lines · Thomas Wei�schuh
Quoted 3 lines · Zhangjin Wu
Quoted 2 lines

The kernel of some architectures can not poweroff qemu-system normally, especially for tinyconfig.

(...)

Quoted 1 line

This feels fairly hacky.

and totally unmaintainable in the long term. It may even fail for some users having localization.

Agree very much in some degree, especially about some of the new added expected string, although I'm carefully choose some. but it is possible to restore the first version and shrink it as possible as we can.

Perhaps I still not explained the background clearly, let me try again to describe ;-)

The added more expected string are used to fill the gap when without output to screen (as suggested by Willy, I have dropped the "print running log to screen" change), quiet output is not friendly to a potential hang and even amplify the influence of a hang experience, but these more expected string did make things more complicated and worse:

    TIMEOUT_CMD = t=$(QEMU_TIMEOUT); past=0; err=""; bios=0; kernel=0; init=0; test=0; poweredoff=0;                                                   \
	bios_timeout=$$(expr $$t - 7); kernel_timeout=$$(expr $$t - 5); init_timeout=$$(expr $$t - 3); test_timeout=$$(expr $$t - 1);              \
	echo "Running $(IMAGE_NAME) on qemu-system-$(QEMU_ARCH)";                                                                                  \
	while [ $$t -gt 0 ]; do                                                                                                                    \
	    sleep 1; t=$$(expr $$t - 1); past=$$(expr $$past + 1);                                                                                 \
	    if [ $$bios -eq 0 ] && grep -E "Linux version|Kernel command line|printk: console" "$(RUN_OUT)"; then bios=1; fi;                      \
	    if [ $$bios -eq 1 -a $$kernel -eq 0 ] && grep -E "Run .* as init process" "$(RUN_OUT)"; then kernel=1; fi;                             \
	    if [ $$kernel -eq 1 -a $$init -eq 0 ] && grep -E "Running test" "$(RUN_OUT)"; then init=1; fi;                                         \
	    if [ $$init -eq 1 -a $$test -eq 0 ] && grep -E "Leaving init with final status|Total number of errors" "$(RUN_OUT)"; then test=1; fi;  \
	    if [ $$init -eq 1 ] && grep -E "Kernel panic - not syncing: Attempted to kill init" "$(RUN_OUT)"; then err="test"; sleep 1; break; fi; \
	    if [ $$test -eq 1 ] && grep -E "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then poweredoff=1; sleep 1; break; fi;         \
	    if [ $$past -gt $$bios_timeout -a $$bios -eq 0 ]; then err="bios"; break; fi;                                                          \
	    if [ $$past -gt $$kernel_timeout -a $$kernel -eq 0 ]; then err="kernel"; break; fi;                                                    \
	    if [ $$past -gt $$init_timeout -a $$init -eq 0 ]; then err="init"; break; fi;                                                          \
	    if [ $$past -gt $$test_timeout -a $$test -eq 0 ]; then err="test"; break; fi;                                                          \
	done;                                                                                                                                      \
	if [ -z "$$err" -a $$poweredoff -eq 0 ]; then err="qemu-system-$(QEMU_ARCH)"; fi;                                                          \
	if [ -n "$$err" ]; then echo "$$err may timeout, test failed"; tail -10 $(RUN_OUT); else echo "powered off, test finish"; fi;              \
	pkill -15 qemu-system-$(QEMU_ARCH) || true

The new added lines are just want to keep the console active to tell users the test is running at background, even there is potential hang, users will learn what the progress it is, but if we allow print the running log to screen (may still have the issues mentioned by Willy in another email thread), these lines can be simply removed to get a cleaner version.

The old version didn't add such logic, beside timeout logic, it simply try to match the expected 'reboot: ' string, but we do need to match the 'Kernel panic' string too to catch a crash nolibc-test case, so, a simplified version looks like this:

    TIMEOUT_CMD = t=$(QEMU_TIMEOUT); finish=0;                                                                               \
        echo "Running $(IMAGE_NAME) on qemu-system-$(QEMU_ARCH)";                                                            \
	while [ $$t -gt 0 ]; do                                                                                              \
	    sleep 1; t=$$(expr $$t - 1);                                                                                     \
	    grep -E "reboot: System halted|reboot: Power down|Kernel panic" "$(RUN_OUT)"; then finish=1; sleep 1; break; fi; \
	done;                                                                                                                \
	if [ $$finish -eq 1 ]; then echo "test finish"; else echo "test timeout"; fi;                                        \
	pkill -15 qemu-system-$(QEMU_ARCH) || true

Now, the lines are not too many, and the left expected strings should be stable enough, do you like this one? I do expect the timeout command has a "-m" option like this:

    timeout --foreground 10 -m "reboot: |Kernel panic" qemu-system-$(QEMU_ARCH) ...
Quoted 2 lines
Quoted 2 lines

Before we complicated nolibc-test to handle the no-procfs case to save a few seconds building the kernel and now we have fairly big timeouts.

In reality, the answer is NO to "now we have fairly big timeouts".

Firstly, If there is no hang or no poweroff failure, every run will quit normally.

Secondly, even there is a poweroff failure or kernel panic, a 'reboot: ' line will be always printed as expected and quit normally.

Thirdly, just only when there is a hang (not specific to tinyconfig), such as wrong bios version or missing firmware or even kernel hang with new changes (for example, the irqstack related riscv kernel hang I reported), blabla. such hangs will be detected by the fixed timeout value (like a watchdog).

When poweroff fails on a target qemu, it still prints the 'reboot: ' line as below, but will hang there after this line, our timeout logic will detect this line and tell qemu quit as a normal poweroff.

Total number of errors: 0
Leaving init with final status: 0
reboot: System halted /* This line which be printed when reboot() issues, not depends on qemu or kernel driver */
---> detecting reboot: System halted|reboot: Power down|Kernel panic - not syncing: Attempted to kill init|Rebooting ...
test finish /* Even qemu not poweroff successfully, we will kill it while the above line detected */
LOG: Boot run successfully.
Running boot-finish

The oldest method I used locally is the 'timeout' command itself only, it is really a dead wait for a fixed timeout, the new method we used here to match the expected string does help a lot to simulate a always sucessful qemu poweroff support, no kernel and qemu dependent, lightweight enough.

Quoted 9 lines
Quoted 6 lines

And a statemachine that relies on the specific strings emitted by the testsuite.

I would like to get back to something more deterministic and obvious, even at the cost of some time spent compiling the test kernels. (saying this as somebody developing on a 2016 ultrabook)

Agreed!

Is the above explaination clearer? ;-)

Based on the kernel versions from v2.6.10 - v6.x which I have used on more than 7 architectures in my own Linux Lab project, the poweroff support of qemu + kernel is never deterministic, some versions ok, some versions fail, even with the defconfig, that's why an external timeout is always required. I used the raw timeout command + a fixed timeout before, but after we have this patch, I have thought about the expected string logic and it does help to aviod a dead fixed-time wait.

Quoted 8 lines
Quoted 8 lines

"Since the low-level poweroff support is heavily kernel & qemu dependent"

The kernel we can control.

How common are qemus with that are missing poweroff support? As this worked before I guess the only architecture where this could pose a problem would be ppc.

Yes, as explained above, based on the experience I have on tons of kernel versions of different architecture, it is really hard to make poweroff work as expected all the time, as the kernel and qemu changes, it may fail at any version randomly.

Beside ppc, all of my local tinyconfig config files of every architecture are ready for boot and print and also of course for the 'reboot: ' line print. but it is 'hard' to find and enable the left options to just further enable 'poweroff' support.

Firstly, I have tried to enable some of them, but it deviates the tinyconfig goal, for example, x86 requires to enable both ACPI and PCI to just let poweroff work, so, I'm not planning to really enable them.

Secondly, the time cost to just find and enable the poweroff options for every architecture (and even for the new nolibc portings) is huge, I give up after several tries, and they may fail in some future versions randomly, I do think we may be not really interested in fixing up such bugs in kernel drivers side ;-)

Thirdly, as Thomas mentioned before, the wireguard test use tinyconfig too, just found it also gives up the poweroff support in its config options and it use a simple raw timeout command, but the timeout is really 'huge', 20m v.s. 10 seconds ;-)

    grep timeout tools/testing/selftests/wireguard/qemu/Makefile
	timeout --foreground 20m qemu-system-$(QEMU_ARCH) \
	timeout --foreground 20m $< \
Quoted 8 lines
Quoted 8 lines

An alternative I would like to put up for discussion:

qemu could provide a watchdog device that is pinged by nolibc-test for each testcase.
After nolibc-test is done and didn't poweroff properly the watchdog will reset the machine. ( -watchog-action poweroff ).

Quoted 5 lines
Quoted 2 lines

The disadvantages are that we would need to add watchdog drivers to the kernels and figure out the correct watchdog devices and drivers for each arch.

It's an interesting idea, though at first glance it does not seem to have one for PPC.

Good idea, I even asked one of the QEMU maintainers whether is possible to add something like -boot-timeout option to QEMU and let qemu quit in a target timeout after start, but it may also function as a dead fixed-time wait.

qemu-system-xxx -boot-timeout <TIMEOUT>

|<-----------------<TIMEOUT>---------------------->

^ ^ ^
qemu start | qemu timeout and quit
|
power off string may be detected here and quit normally

Quoted 9 lines

I think I have a much simpler idea: we don't care about PPC32. I mean OK it can be supported if it happens to work, we will just not include it in default runs, because it will require Ctrl-C to finish, and so what ? nolibc has been in the kernel for 5 years or so, nobody ever cared about PPC, why should we suddenly break or complicate everything just to support a sub-arch that nobody found interesting to add till now?

Yes, this timeout logic is ok to be removed from this patchset, till we get a better solution.

Quoted 4 lines
Quoted 1 line

It seems virtio-watchdog is not yet usable.

Then it might become an option for the future when it eventually works.

Thanks,
Zhangjin

Quoted 2 lines

Thanks,
Willy

maintainer
#17 lore raw

Re: [PATCH v3 08/12] selftests/nolibc: allow quit qemu-system when poweroff fails

On Sat, Jul 29, 2023 at 08:05:00PM +0800, Zhangjin Wu wrote:
(...)

Quoted 21 lines

The new added lines are just want to keep the console active to tell users the test is running at background, even there is potential hang, users will learn what the progress it is, but if we allow print the running log to screen (may still have the issues mentioned by Willy in another email thread), these lines can be simply removed to get a cleaner version.

The old version didn't add such logic, beside timeout logic, it simply try to match the expected 'reboot: ' string, but we do need to match the 'Kernel panic' string too to catch a crash nolibc-test case, so, a simplified version looks like this:

    TIMEOUT_CMD = t=$(QEMU_TIMEOUT); finish=0;                                                                               \
        echo "Running $(IMAGE_NAME) on qemu-system-$(QEMU_ARCH)";                                                            \
	while [ $$t -gt 0 ]; do                                                                                              \
	    sleep 1; t=$$(expr $$t - 1);                                                                                     \
	    grep -E "reboot: System halted|reboot: Power down|Kernel panic" "$(RUN_OUT)"; then finish=1; sleep 1; break; fi; \
	done;                                                                                                                \
	if [ $$finish -eq 1 ]; then echo "test finish"; else echo "test timeout"; fi;                                        \
	pkill -15 qemu-system-$(QEMU_ARCH) || true
Quoted 2 lines

Now, the lines are not too many, and the left expected strings should be stable enough, do you like this one?

Do you realize how unreadable and undebuggable this is for the casual reader who just tries to figure what command line to pass to qemu to run the test manually ? You seem to focus a lot on "if the test hangs" but nobody ever reported such a problem in the last few years. However we've been spending weeks discussing how to add support for extra features or architectures and such hacky stuff definitely steps in the way and complicates everything.

Quoted 4 lines

I do expect the timeout command has
a "-m" option like this:

    timeout --foreground 10 -m "reboot: |Kernel panic" qemu-system-$(QEMU_ARCH) ...

I suggest that we give up on this timeout thing completely. You've shown that it significantly complicates everything and we yet have to find a single valid use case. Other subsystems also use QEMU without this. rcutorture does perform some process management but it's way more advanced and complete, and there's a reason: the tests are usually not supposed to end by themselves.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Before we complicated nolibc-test to handle the no-procfs case to save a few seconds building the kernel and now we have fairly big timeouts.

In reality, the answer is NO to "now we have fairly big timeouts".

Firstly, If there is no hang or no poweroff failure, every run will quit normally.

Which is one good reason for not complexifying everything.

Quoted 2 lines

Secondly, even there is a poweroff failure or kernel panic, a 'reboot: ' line will be always printed as expected and quit normally.

This is not supposed to happen in automated tests, and archs that fail at this will simply be either excluded from automated tests, or will be run through whatever external timeout mechanism.

Quoted 5 lines

Thirdly, just only when there is a hang (not specific to tinyconfig), such as wrong bios version or missing firmware or even kernel hang with new changes (for example, the irqstack related riscv kernel hang I reported), blabla. such hangs will be detected by the fixed timeout value (like a watchdog).

These are local tools issues, we can't fix them all and it's not our job. We're just providing an easy and hopefully convenient framework to test syscalls. We're not supposed to require users to have to go through complex debugging in this. And if they face a failure due to their local tools (like I had with my old qemu version), they'll just work around it by rebuilding it and that will be done. In the worst case, some archs might require a Ctrl-C once in a while during a manual test. No big deal. Definitely not as big as spending 10 minutes trying to figure how to find one's way through a complicated makefile, or wondering what's that runaway qemu process in the background that refuses to die, etc.

Quoted 12 lines

When poweroff fails on a target qemu, it still prints the 'reboot: ' line as below, but will hang there after this line, our timeout logic will detect this line and tell qemu quit as a normal poweroff.

Total number of errors: 0
Leaving init with final status: 0
reboot: System halted /* This line which be printed when reboot() issues, not depends on qemu or kernel driver */
---> detecting reboot: System halted|reboot: Power down|Kernel panic - not syncing: Attempted to kill init|Rebooting ...
test finish /* Even qemu not poweroff successfully, we will kill it while the above line detected */
LOG: Boot run successfully.
Running boot-finish

Quoted 5 lines

The oldest method I used locally is the 'timeout' command itself only, it is really a dead wait for a fixed timeout, the new method we used here to match the expected string does help a lot to simulate a always sucessful qemu poweroff support, no kernel and qemu dependent, lightweight enough.

I'm still definitely not fond of it because it turns something super simple into something complex and likely unreliable, for very little benefit.

Quoted 1 line

Is the above explaination clearer? ;-)

At least given the numerous approaches we've seen, now I'm convinced I don't want to see that anymore. Too much complication, a feeling of hackish and unreliable solution that will very likely cause lots of fixes in the future.

Quoted 5 lines

Based on the kernel versions from v2.6.10 - v6.x which I have used on more than 7 architectures in my own Linux Lab project, the poweroff support of qemu + kernel is never deterministic, some versions ok, some versions fail, even with the defconfig, that's why an external timeout is always required.

Again, never ever experienced such a problem with the default configs for the supported archs, with one of my machines having a qemu version as old as 2.7. Paul always runs all the tests as well and never reported this problem either. Thus I would like that we stick to precise facts about problems that occur rather than just papering over them in a generic way.

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

"Since the low-level poweroff support is heavily kernel & qemu dependent"

The kernel we can control.

How common are qemus with that are missing poweroff support? As this worked before I guess the only architecture where this could pose a problem would be ppc.

Quoted 4 lines

Yes, as explained above, based on the experience I have on tons of kernel versions of different architecture, it is really hard to make poweroff work as expected all the time, as the kernel and qemu changes, it may fail at any version randomly.

Please could you provide us with a reproducer for this problem, with the mainline commit ID, arch, toolchain used, qemu version, because I think you're generalizing over a few cases that happened during your tinyconfig tests, for various possible reasons, but which are likely not good reasons for complicating everything.

Quoted 4 lines

Beside ppc, all of my local tinyconfig config files of every architecture are ready for boot and print and also of course for the 'reboot: ' line print. but it is 'hard' to find and enable the left options to just further enable 'poweroff' support.

If tinyconfig is not fixable, it's not usable, period. Right now all archs stop fine for many of us with defconfig. If only a few tinyconfig fail we'd rather invest time on these specific ones trying to figure what options you need to add to the extra_config.

Quoted 9 lines

Firstly, I have tried to enable some of them, but it deviates the tinyconfig goal, for example, x86 requires to enable both ACPI and PCI to just let poweroff work, so, I'm not planning to really enable them.

Secondly, the time cost to just find and enable the poweroff options for every architecture (and even for the new nolibc portings) is huge, I give up after several tries, and they may fail in some future versions randomly, I do think we may be not really interested in fixing up such bugs in kernel drivers side ;-)

OK so better just give up on tinyconfig if it's not suitable for the tests ? The more you present the shortcomings that come with them, the less appealing it looks now.

Quoted 8 lines

Thirdly, as Thomas mentioned before, the wireguard test use tinyconfig too, just found it also gives up the poweroff support in its config options and it use a simple raw timeout command, but the timeout is really 'huge', 20m v.s. 10 seconds ;-)

    grep timeout tools/testing/selftests/wireguard/qemu/Makefile
	timeout --foreground 20m qemu-system-$(QEMU_ARCH) \
	timeout --foreground 20m $< \

As I previously mentioned, I'm not fundamentally against a simple timeout command *iff* we can validate the exact problem and the conditions where it happens and we decide that it's the best workaround. And I suspect that even once we find it we'll prefer to just not run that specific setup by default and that's all.

Quoted 11 lines
Quoted 8 lines

I think I have a much simpler idea: we don't care about PPC32. I mean OK it can be supported if it happens to work, we will just not include it in default runs, because it will require Ctrl-C to finish, and so what ? nolibc has been in the kernel for 5 years or so, nobody ever cared about PPC, why should we suddenly break or complicate everything just to support a sub-arch that nobody found interesting to add till now?

Yes, this timeout logic is ok to be removed from this patchset, till we get a better solution.

Thanks.

It would really help if your series could focus on *one thing* at a time. Currently I feel like in almost all patch series you've sent there are good stuff that could have been merged already, but which are mixed with hacks or unacceptable massive reworks that just result in the rest being kept on hold.

I would really appreciate it if you thought about clearly presenting the problems you're trying to solve before sending patch series, so that we can collectively decide whether these problems deserve being fixed or can be ignored, and if the cost of addressing them outweigh their cost. It would save many hours of review of patches whose goal is not always very clear. A good rule of thumb is that something that is only added and that provides a specific feature generally suffers not much discussion (beyond the usual style/naming etc). But patches that modify existing process, code organization or affect reliability should be discussed and argumented before even being created. It's easier to discuss a purpose than to try to review a patch for a context that is not very clear.

Thanks,
Willy

#18 lore raw

Re: [PATCH v3 08/12] selftests/nolibc: allow quit qemu-system when poweroff fails

Hi, Willy

[...]

Quoted 2 lines

You seem to focus a lot on "if the test hangs"
but nobody ever reported such a problem in the last few years. However

Willy, as Thomas shared in another reply too, there is really a poweroff failure (even with defconfig) in the default ppc/g3beige 32-bit PowerPC Qemu we are adding support for.

Btw, It is possible to switch another 32-bit PowerPC board who support poweroff in qemu+kernel side, but need more investigate and survey, I have tried another two, no lucky, the ppce500 webpage [1] even claims there is 'Power-off functionality via one GPIO pin', but tried to enable the related gpios and reset options, still not work, perhaps I need help from more PowerPc users, will ask somebody to help me ;-)

[1]: https://qemu.readthedocs.io/en/latest/system/ppc/ppce500.html

Quoted 3 lines

we've been spending weeks discussing how to add support for extra features or architectures and such hacky stuff definitely steps in the way and complicates everything.

The background is, based on your feedback in another reply, printing the log to screen (another patch I have removed in this revision) may have some issues before. But if without any log and just hang after a poweroff failure and even further no timeout ... then, dead hang, that is really hard to debug, edit of Makefile and rerun and then we will know it fails at poweroff or just during qemu start ..., as you replied below, CTRL+C may help to cat the log file too, but it waste a CTRL+C + cat during the early development stage, especially when we require frequenty modify and test.

[...]

Quoted 5 lines

This is not supposed to happen in automated tests, and archs that fail at this will simply be either excluded from automated tests, or will be run through whatever external timeout mechanism.

Agree with external timeout mechanism or even excluded from automated tests, but we at least to allow to do basic test without editing the Makefile or without forever Ctrl-C + cat run.out manually ;-)

So, what about at least restore part of my old 'print running log to screen' patch like this:

    # run the tests after building the kernel
    run: kernel
    	$(Q)$(QEMU_SYSTEM_RUN) | tee "$(RUN_OUT)"
    	$(Q)$(REPORT) "$(RUN_OUT)"
    # re-run the tests from an existing kernel
    rerun:
    	$(Q)$(QEMU_SYSTEM_RUN) | tee "$(RUN_OUT)"
    	$(Q)$(REPORT) "$(RUN_OUT)"

Or even better, allow use a swtich to control it (the one we also removed from an old patch).

    ifneq ($(QUIET_RUN),1)
    LOG_OUT= | tee "$(RUN_OUT)"
    else
    LOG_OUT= > "$(RUN_OUT)"
    endif

If missing this, the early stages of a new architecture porting is definitely very hard, not think about poweroff failure, just think about some other potential exceptions which simply stop the booting or testing (not continue and not exit).

Quoted 6 lines
Quoted 5 lines

Thirdly, just only when there is a hang (not specific to tinyconfig), such as wrong bios version or missing firmware or even kernel hang with new changes (for example, the irqstack related riscv kernel hang I reported), blabla. such hangs will be detected by the fixed timeout value (like a watchdog).

Quoted 11 lines

These are local tools issues, we can't fix them all and it's not our job. We're just providing an easy and hopefully convenient framework to test syscalls. We're not supposed to require users to have to go through complex debugging in this. And if they face a failure due to their local tools (like I had with my old qemu version), they'll just work around it by rebuilding it and that will be done. In the worst case, some archs might require a Ctrl-C once in a while during a manual test. No big deal. Definitely not as big as spending 10 minutes trying to figure how to find one's way through a complicated makefile, or wondering what's that runaway qemu process in the background that refuses to die, etc.

Again Willy, without log and without timeout (with a tail of last 10 lines in the old version), just a 'forever' hang, Ctrl-C helps but not much, still require to check the log to see what happens or editing the Makefile to enable the logging temply, it is realy time-waste during development ;-)

[...]

Quoted 8 lines

Again, never ever experienced such a problem with the default configs for the supported archs, with one of my machines having a qemu version as old as 2.7. Paul always runs all the tests as well and never reported this problem either. Thus I would like that we stick to precise facts about problems that occur rather than just papering over them in a generic way.

Willy, before, the old versions of this patchset did only configure QEMU_TIMEOUT for the failed 32-bit PowerPC and not enabled for the other boards, but later after a dicussion (perhaps I misunderstood one of your suggestions?), this revision configure a default one for all, I also thought about it is not good for all boards, but still a choice to a board really have no poweroff support (external one is ok if with explicitly log printing to screen).

we have used something like this before:

    ifneq ($(QEMU_TIMEOUT),)
    ...
    endif
Quoted 14 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

"Since the low-level poweroff support is heavily kernel & qemu dependent"

The kernel we can control.

How common are qemus with that are missing poweroff support? As this worked before I guess the only architecture where this could pose a problem would be ppc.

Quoted 4 lines

Yes, as explained above, based on the experience I have on tons of kernel versions of different architecture, it is really hard to make poweroff work as expected all the time, as the kernel and qemu changes, it may fail at any version randomly.

Quoted 6 lines

Please could you provide us with a reproducer for this problem, with the mainline commit ID, arch, toolchain used, qemu version, because I think you're generalizing over a few cases that happened during your tinyconfig tests, for various possible reasons, but which are likely not good reasons for complicating everything.

It is very hard to list all, the following one is a section [1] (not updated for a long time) of the document of my 'Linux Lab' project:

### 6.2.2 Poweroff hang

Both of the poweroff and reboot commands not work on these boards currently (LINUX=v5.1):

* mipsel/malta (exclude LINUX=v2.6.36)
* mipsel/ls232
* mipsel/ls1b
* mips64el/ls2k
* mips64el/ls3a7a
* aarch64/raspi3
* arm/versatilepb

System will directly hang there while running poweroff or reboot, to exit qemu, please pressing CTRL+a x or using pkill qemu.

To test such boards automatically, please make sure setting TEST_TIMEOUT, e.g. make test TEST_TIMEOUT=50.

Welcome to fix up them.

IMHO, since my project is mainly for kernel learning and development, I even don't care about why it not poweroff successfully at last although I have tried my best to find the realted options but failed at last, the reasons may be complex, from kernel side or from qemu side, both possible, or even simply a kernel option changed and the config options are not updated timely.

We have so many boards with kernel version from v2.6.10 to v6.4, many poweroff failures before:

    $ ls -1 -d */*
    aarch64/raspi3
    aarch64/virt
    arm/ebf-imx6ull
    arm/mcimx6ul-evk
    arm/versatilepb
    arm/vexpress-a9
    arm/virt
    i386/pc
    loongarch64/virt
    mips64el/loongson3-virt
    mips64el/ls2k
    mips64el/ls3a7a
    mipsel/ls1b
    mipsel/ls232
    mipsel/malta
    ppc64le/powernv
    ppc64le/pseries
    ppc64/powernv
    ppc64/pseries
    ppc/g3beige
    ppc/ppce500
    riscv32/virt
    riscv64/virt
    s390x/s390-ccw-virtio
    x86_64/pc
    The kernel we used for i386/pc:

    v2.6.10
    v2.6.11.12
    v2.6.12.6
    v2.6.21.5
    v2.6.24.7
    v2.6.34.9
    v2.6.35.14
    v2.6.36
    v3.10.108
    v4.6.7
    v5.1
    v5.13
    v5.2
    v6.1.1

[1]: https://github.com/tinyclub/linux-lab/blob/master/README.md#622-poweroff-hang

Quoted 10 lines
Quoted 4 lines

Beside ppc, all of my local tinyconfig config files of every architecture are ready for boot and print and also of course for the 'reboot: ' line print. but it is 'hard' to find and enable the left options to just further enable 'poweroff' support.

If tinyconfig is not fixable, it's not usable, period. Right now all archs stop fine for many of us with defconfig. If only a few tinyconfig fail we'd rather invest time on these specific ones trying to figure what options you need to add to the extra_config.

Yes, that's why I plan to send the left patches by architecture, it will give us more time to configure and confirm the missing poweroff options if really required, I will ask somebody else to work with me together.

Quoted 10 lines
Quoted 9 lines

Firstly, I have tried to enable some of them, but it deviates the tinyconfig goal, for example, x86 requires to enable both ACPI and PCI to just let poweroff work, so, I'm not planning to really enable them.

Secondly, the time cost to just find and enable the poweroff options for every architecture (and even for the new nolibc portings) is huge, I give up after several tries, and they may fail in some future versions randomly, I do think we may be not really interested in fixing up such bugs in kernel drivers side ;-)

Quoted 4 lines

OK so better just give up on tinyconfig if it's not suitable for the tests ? The more you present the shortcomings that come with them, the less appealing it looks now.

The only shortcoming is some 'poweroff' failures currently, I do think the time-saved is huge, with tinyconfig, I even don't want to try the time-consumer defconfig anymore (although it is a requirement of many kernel features for the last release), especially for nolibc development ;-)

I'm using tinyconfig+nolibc for most of the boards supported by my Linux Lab project, I'm really happy with it, I use similar timeout + expected string logic now, it works better than the old pure timeout logic, then, I don't care about which poweroff options the target board want, just ignore the requirement. It is a very good kernel learning and development experience, I have used a defconfig based config and also a small config customized for nolibc for them before, but with the new tinyconfig (extra boot/print options) suggested by Thomas, it is really a good and a whole new experience.

To be honest, from my side, it is ok to add tinyconfig for nolibc or not, because I can use the test script from Linux Lab project to test new nolibc features, but I do think this will help a lot for both the kernel and nolibc developers, so, I'm happy to continue, even we are facing some real difficulties currently.

[...]

Quoted 5 lines
Quoted 4 lines
Quoted 1 line

Yes, this timeout logic is ok to be removed from this patchset, till we get a better solution.

But again as explained above, we do need something to cope with 'dead' hang without any output.

Quoted 8 lines

Thanks.

It would really help if your series could focus on *one thing* at a time. Currently I feel like in almost all patch series you've sent there are good stuff that could have been merged already, but which are mixed with hacks or unacceptable massive reworks that just result in the rest being kept on hold.

Agree, thanks, I will try my best to split a huge stuff to smaller ones, I was a little hurried to want to upstream the non-rv32 related parts quickly, then, we can back to rv32 asap ;-)

Quoted 6 lines

I would really appreciate it if you thought about clearly presenting the problems you're trying to solve before sending patch series, so that we can collectively decide whether these problems deserve being fixed or can be ignored, and if the cost of addressing them outweigh their cost. It would save many hours of review of patches whose goal is not always very clear.

I like this suggestion, since you have mentioned before, based on your suggestions, like the syscall.h stuff, a disucss or a proposal is fired before a real patchset, although I think it worths a real RFC patchset, but let's delay it and I do need more work to solve the questions asked from you, this may be also related to some ideas mentioned by Arnd before, seems he have planed to generate some syscall templates from the kernel .tbls, let's discuss this in another thread, please ignore this one.

Quoted 7 lines

A good rule of thumb is that something that
is only added and that provides a specific feature generally suffers not much discussion (beyond the usual style/naming etc). But patches that modify existing process, code organization or affect reliability should be discussed and argumented before even being created. It's easier to discuss a purpose than to try to review a patch for a context that is not very clear.

That's reasonable, a good rule is 'subtraction', not 'addition', sometimes, the implementation of new idea does add more impulsion to modify anything, it require calmness to go back to the core issue we want to solve.

Thanks a lot.

Best regards,
Zhangjin

Quoted 3 lines

Thanks,
Willy

#19 lore raw

PATCH v3 9/12 selftests/nolibc: customize QEMU_TIMEOUT for ppc64/ppc64le

In reply to Zhangjin Wu

Both ppc64 and ppc64le use slow bios in some qemu versions, let's increase the timeout to make sure the running qemu would not be killed before the test finish.

---
 tools/testing/selftests/nolibc/Makefile | 2 ++
 1 file changed, 2 insertions(+)
tools/testing/selftests/nolibc/Makefile +2 -0
index 9a57de3b283c..ad2538ec5eb0 100644
@@ -106,6 +106,8 @@ QEMU_ARGS_loongarch  = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=N
 QEMU_ARGS            = $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_EXTRA)
 
 # QEMU_TIMEOUT: some architectures can not poweroff normally, especially for tinyconfig
+QEMU_TIMEOUT_ppc64     = 15
+QEMU_TIMEOUT_ppc64le   = 25
 QEMU_TIMEOUT           = $(or $(QEMU_TIMEOUT_$(XARCH)),10)
 
 # OUTPUT is only set when run from the main makefile, otherwise
Signature · 1 line
2.25.1
#20 lore raw

PATCH v3 10/12 selftests/nolibc: tinyconfig: add extra common options

In reply to Zhangjin Wu

The tinyconfig target from top-level Makefile has already enabled some common options, but they are not enough to enable boot and print.

    $ find kernel/ arch/*/ -name "tiny*.config"
    kernel/configs/tiny-base.config
    kernel/configs/tiny.config
    arch/x86/configs/tiny.config

To enable qemu boot and console print, additional kernel config options are required, include the common parts and the architecture specific parts.

Here adds minimal extra common parts for all architectures:

* for initrd: CONFIG_BLK_DEV_INITRD
* for init executable: CONFIG_BINFMT_ELF
* for test result print: CONFIG_PRINTK, CONFIG_TTY

---
 tools/testing/selftests/nolibc/configs/common.config | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 tools/testing/selftests/nolibc/configs/common.config
tools/testing/selftests/nolibc/configs/common.config new file +4 -0
new file mode 100644
index 000000000000..3957f812faac
@@ -0,0 +1,4 @@
+CONFIG_BLK_DEV_INITRD=y
+CONFIG_BINFMT_ELF=y
+CONFIG_PRINTK=y
+CONFIG_TTY=y
Signature · 1 line
2.25.1
#21 lore raw

PATCH v3 11/12 selftests/nolibc: tinyconfig: add support for 32/64-bit powerpc

In reply to Zhangjin Wu

This adds extra config options for ppc, ppc64le and ppc64, now, it is able to use tinyconfig as the minimal config target to speed up the run target of powerpc:

    $ for arch in ppc ppc64 ppc64le; do \
        mkdir -p $PWD/kernel-$arch; \
        make defconfig run DEFCONFIG=tinyconfig ARCH=$arch O=$PWD/kernel-$arch | grep status ; \
      done

rerun with architecture specific run.out:

    $ for arch in ppc ppc64 ppc64le; do \
        mkdir -p $PWD/kernel-$arch; \
        make rerun ARCH=$arch O=$PWD/kernel-$arch RUN_OUT=$PWD/run.$arch.out | grep status ; \
      done

report with existing test log:

    $ for arch in powerpc powerpc64 powerpc64le; do \
        make report RUN_OUT=$PWD/run.$arch.out | grep status ; \
      done
---
 tools/testing/selftests/nolibc/configs/ppc.config     | 3 +++
 tools/testing/selftests/nolibc/configs/ppc64.config   | 3 +++
 tools/testing/selftests/nolibc/configs/ppc64le.config | 4 ++++
 3 files changed, 10 insertions(+)
 create mode 100644 tools/testing/selftests/nolibc/configs/ppc64.config
 create mode 100644 tools/testing/selftests/nolibc/configs/ppc64le.config
tools/testing/selftests/nolibc/configs/ppc.config +3 -0
index b1975f8253f7..29123cee14c4 100644
@@ -1,3 +1,6 @@
+CONFIG_COMPAT_32BIT_TIME=y
+CONFIG_PPC_PMAC=y
+CONFIG_PPC_OF_BOOT_TRAMPOLINE=y
 CONFIG_SERIAL_PMACZILOG=y
 CONFIG_SERIAL_PMACZILOG_TTYS=y
 CONFIG_SERIAL_PMACZILOG_CONSOLE=y
tools/testing/selftests/nolibc/configs/ppc64.config new file +3 -0
new file mode 100644
index 000000000000..4e17f0cdb99f
@@ -0,0 +1,3 @@
+CONFIG_PPC64=y
+CONFIG_PPC_POWERNV=y
+CONFIG_HVC_OPAL=y
tools/testing/selftests/nolibc/configs/ppc64le.config new file +4 -0
new file mode 100644
index 000000000000..713b227f506f
@@ -0,0 +1,4 @@
+CONFIG_PPC64=y
+CONFIG_PPC_POWERNV=y
+CONFIG_HVC_OPAL=y
+CONFIG_CPU_LITTLE_ENDIAN=y
Signature · 1 line
2.25.1
#22 lore raw

PATCH v3 12/12 selftests/nolibc: speed up some targets with multiple jobs

In reply to Zhangjin Wu

The sysroot install and kernel build targets are time cost, let's use -j<N> to parallelize them with multiple jobs.

---
 tools/testing/selftests/nolibc/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/nolibc/Makefile +2 -2
index ad2538ec5eb0..1b45c22f9a94 100644
@@ -171,7 +171,7 @@ PHONY = sysroot/$(ARCH)/include
 sysroot/$(ARCH)/include:
 	$(Q)rm -rf sysroot/$(ARCH) sysroot/sysroot
 	$(QUIET_MKDIR)mkdir -p sysroot
-	$(Q)$(MAKE) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone
+	$(Q)$(MAKE) -j$$(nproc) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone
 	$(Q)mv sysroot/sysroot sysroot/$(ARCH)
 
 ifneq ($(NOLIBC_SYSROOT),0)
@@ -211,7 +211,7 @@ initramfs: nolibc-test
 	$(Q)cp nolibc-test initramfs/init
 
 # common macros for kernel targets
-MAKE_KERNEL   = $(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
+MAKE_KERNEL   = $(MAKE) -j$$(nproc) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
 KERNEL_CONFIG = $(objtree)/.config
 KERNEL_IMAGE  = $(objtree)/$(IMAGE)
 
Signature · 1 line
2.25.1
#23 lore raw

Re: [PATCH v3 12/12] selftests/nolibc: speed up some targets with multiple jobs

Quoted 13 lines · Zhangjin Wu

The sysroot install and kernel build targets are time cost, let's use -j<N> to parallelize them with multiple jobs.

---
 tools/testing/selftests/nolibc/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/nolibc/Makefile +0 -0
index ad2538ec5eb0..1b45c22f9a94 100644

tools/testing/selftests/nolibc/Makefile PHONY = sysroot/$(ARCH)/include · line 174

Quoted 5 lines
tools/testing/selftests/nolibc/Makefile +1 -1
@@ -171,7 +171,7 @@ PHONY = sysroot/$(ARCH)/include
 sysroot/$(ARCH)/include:
 	$(Q)rm -rf sysroot/$(ARCH) sysroot/sysroot
 	$(QUIET_MKDIR)mkdir -p sysroot
-	$(Q)$(MAKE) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone
+	$(Q)$(MAKE) -j$$(nproc) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone

This should already work when the users specify -j on the make command line themselves.
I'm not a fan of force-enabling it here.

Quoted 15 lines
tools/testing/selftests/nolibc/Makefile +1 -1
 	$(Q)mv sysroot/sysroot sysroot/$(ARCH)
 
 ifneq ($(NOLIBC_SYSROOT),0)
@@ -211,7 +211,7 @@ initramfs: nolibc-test
 	$(Q)cp nolibc-test initramfs/init
 
 # common macros for kernel targets
-MAKE_KERNEL   = $(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
+MAKE_KERNEL   = $(MAKE) -j$$(nproc) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
 KERNEL_CONFIG = $(objtree)/.config
 KERNEL_IMAGE  = $(objtree)/$(IMAGE)
 
Signature · 1 line
2.25.1
maintainer
#24 lore raw

Re: [PATCH v3 12/12] selftests/nolibc: speed up some targets with multiple jobs

Quoted 9 lines · Thomas Weißschuh
Quoted 8 lines · Zhangjin Wu

The sysroot install and kernel build targets are time cost, let's use -j<N> to parallelize them with multiple jobs.

---
 tools/testing/selftests/nolibc/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 14 lines

tools/testing/selftests/nolibc/Makefile PHONY = sysroot/$(ARCH)/include · line 174

Quoted 10 lines
tools/testing/selftests/nolibc/Makefile +1 -1
index ad2538ec5eb0..1b45c22f9a94 100644
@@ -171,7 +171,7 @@ PHONY = sysroot/$(ARCH)/include
 sysroot/$(ARCH)/include:
 	$(Q)rm -rf sysroot/$(ARCH) sysroot/sysroot
 	$(QUIET_MKDIR)mkdir -p sysroot
-	$(Q)$(MAKE) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone
+	$(Q)$(MAKE) -j$$(nproc) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone

This should already work when the users specify -j on the make command line themselves.
I'm not a fan of force-enabling it here.

Indeed, we must not do that, because some users might for instance prefer to build multiple archs in parallel and benefit from a better parallelism and now they'd end up with too many processes.

Willy

#25 lore raw

Re: [PATCH v3 12/12] selftests/nolibc: speed up some targets with multiple jobs

Quoted 25 lines · Thomas Wei�schuh
Quoted 9 lines · Zhangjin Wu
Quoted 8 lines

The sysroot install and kernel build targets are time cost, let's use -j<N> to parallelize them with multiple jobs.

---
 tools/testing/selftests/nolibc/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 14 lines

tools/testing/selftests/nolibc/Makefile PHONY = sysroot/$(ARCH)/include · line 174

Quoted 10 lines
tools/testing/selftests/nolibc/Makefile +1 -1
index ad2538ec5eb0..1b45c22f9a94 100644
@@ -171,7 +171,7 @@ PHONY = sysroot/$(ARCH)/include
 sysroot/$(ARCH)/include:
 	$(Q)rm -rf sysroot/$(ARCH) sysroot/sysroot
 	$(QUIET_MKDIR)mkdir -p sysroot
-	$(Q)$(MAKE) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone
+	$(Q)$(MAKE) -j$$(nproc) -C ../../../include/nolibc ARCH=$(ARCH) OUTPUT=$(CURDIR)/sysroot/ headers_standalone

This should already work when the users specify -j on the make command line themselves.
I'm not a fan of force-enabling it here.

Quoted 3 lines

Indeed, we must not do that, because some users might for instance prefer to build multiple archs in parallel and benefit from a better parallelism and now they'd end up with too many processes.

Ok, let users do what they want.

Zhangjin

Quoted 2 lines

Willy