Skip to contentSkip to conversation

Add barebox bootloader support (and testing)

PATCH Superseded Enrico Jorns 4 participants 23 messages Mar 31, 2023 – Apr 24, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Add barebox bootloader support (and testing)

This adds support for the barebox bootloader (and tools) to oe-core.

In order to have proper testing, this extends oe-selftest to allow basic testing of bootloaders. While at it, cover both barebox and u-boot.

v2:
* set myself as maintainer in maintainers.inc
* move doc from documentation.conf to recipe
* update to barebox v2023.03 (including fixes for musl, etc.)
* set standard configs for qemu machines to allow building them
* add better efi dir support in barebox recipe
* enable testing bootloaders in oeqa
* add test cases for barebox (and u-boot)

Enrico Jorns (7):
barebox: set default BAREBOX_CONFIG for qemu machines oeqa/utils/qemurunner: support ignoring vt100 escape sequences oeqa/utils/qemurunner: simplify output parsing and make
crlf-compatible
oeqa/utils/commands: document runqemu context manager
oeqa: support passing custom boot patterns to runqemu oeqa/selftest/cases: add barebox tests oeqa/selftest/cases: add basic u-boot test

Marco Felsch (2):
barebox: add initial support
barebox-tools: add initial barebox tools support

 meta/conf/distro/include/maintainers.inc      |   2 +
 meta/lib/oeqa/selftest/cases/barebox.py       |  72 +++++++++
 meta/lib/oeqa/selftest/cases/uboot.py         |  40 +++++
 meta/lib/oeqa/targetcontrol.py                |   5 +-
 meta/lib/oeqa/utils/commands.py               |  22 ++-
 meta/lib/oeqa/utils/qemurunner.py             |  37 +++--
 .../barebox/barebox-tools_2023.03.0.bb        |  56 +++++++
 meta/recipes-bsp/barebox/barebox.inc          | 138 ++++++++++++++++++
 meta/recipes-bsp/barebox/barebox_2023.03.0.bb |   3 +
 9 files changed, 351 insertions(+), 24 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/barebox.py
 create mode 100644 meta/lib/oeqa/selftest/cases/uboot.py
 create mode 100644 meta/recipes-bsp/barebox/barebox-tools_2023.03.0.bb
 create mode 100644 meta/recipes-bsp/barebox/barebox.inc
 create mode 100644 meta/recipes-bsp/barebox/barebox_2023.03.0.bb
Signature · 1 line
2.39.2
#1 lore raw

PATCH v2 1/9 barebox: add initial support

From: Marco Felsch <[email protected]>

This adds the support for the barebox bootloader to oe-core. The recipe is based on the recipe found in meta-ptx [1] with a few minor adaptions.

This basic support includes the bootloader and the target tools to interact with the bootloader. The host tools support is not part of this commit. This will be added later on as separate recipe.

[1] https://github.com/pengutronix/meta-ptx/tree/master/recipes-bsp/barebox

---
 meta/conf/distro/include/maintainers.inc      |   1 +
 meta/recipes-bsp/barebox/barebox.inc          | 133 ++++++++++++++++++
 meta/recipes-bsp/barebox/barebox_2023.03.0.bb |   3 +
 3 files changed, 137 insertions(+)
 create mode 100644 meta/recipes-bsp/barebox/barebox.inc
 create mode 100644 meta/recipes-bsp/barebox/barebox_2023.03.0.bb
meta/conf/distro/include/maintainers.inc +1 -0
index 682ec2cfdf..86d5519489 100644
@@ -54,6 +54,7 @@ RECIPE_MAINTAINER:pn-automake = "Robert Yang <[email protected]>"
 RECIPE_MAINTAINER:pn-avahi = "Yi Zhao <[email protected]>"
 RECIPE_MAINTAINER:pn-babeltrace = "Alexander Kanavin <[email protected]>"
 RECIPE_MAINTAINER:pn-babeltrace2 = "Alexander Kanavin <[email protected]>"
+RECIPE_MAINTAINER:pn-barebox = "Enrico Jörns <[email protected]>"
 RECIPE_MAINTAINER:pn-baremetal-helloworld = "Alejandro Hernandez <[email protected]>"
 RECIPE_MAINTAINER:pn-base-files = "Anuj Mittal <[email protected]>"
 RECIPE_MAINTAINER:pn-base-passwd = "Anuj Mittal <[email protected]>"
meta/recipes-bsp/barebox/barebox.inc new file +133 -0

133 lines not on this page · raw message

meta/recipes-bsp/barebox/barebox_2023.03.0.bb new file +3 -0
new file mode 100644
index 0000000000..860b68a95f
@@ -0,0 +1,3 @@
+require barebox.inc
+
+SRC_URI[sha256sum] = "d0f78a69ba240327247c8fd0e1d45287e4a0dff99ed847e9a696cc2da0cf388c"
Signature · 1 line
2.39.2
#2 lore raw

PATCH v2 2/9 barebox-tools: add initial barebox tools support

In reply to Enrico Jorns

From: Marco Felsch <[email protected]>

Add initial support to build the barebox tools for the host and the target.

---
 meta/conf/distro/include/maintainers.inc      |  1 +
 .../barebox/barebox-tools_2023.03.0.bb        | 56 +++++++++++++++++++
 2 files changed, 57 insertions(+)
 create mode 100644 meta/recipes-bsp/barebox/barebox-tools_2023.03.0.bb
meta/conf/distro/include/maintainers.inc +1 -0
index 86d5519489..1a0ba10d3b 100644
@@ -55,6 +55,7 @@ RECIPE_MAINTAINER:pn-avahi = "Yi Zhao <[email protected]>"
 RECIPE_MAINTAINER:pn-babeltrace = "Alexander Kanavin <[email protected]>"
 RECIPE_MAINTAINER:pn-babeltrace2 = "Alexander Kanavin <[email protected]>"
 RECIPE_MAINTAINER:pn-barebox = "Enrico Jörns <[email protected]>"
+RECIPE_MAINTAINER:pn-barebox-tools = "Enrico Jörns <[email protected]>"
 RECIPE_MAINTAINER:pn-baremetal-helloworld = "Alejandro Hernandez <[email protected]>"
 RECIPE_MAINTAINER:pn-base-files = "Anuj Mittal <[email protected]>"
 RECIPE_MAINTAINER:pn-base-passwd = "Anuj Mittal <[email protected]>"
meta/recipes-bsp/barebox/barebox-tools_2023.03.0.bb new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.39.2
#3 lore raw

PATCH v2 3/9 barebox: set default BAREBOX_CONFIG for qemu machines

In reply to Enrico Jorns

These are set in the barebox recipe rather in the corresponding machines (where they would belong otherwise) to keep the impact of barebox to oe-core minimal for now.

---
 meta/recipes-bsp/barebox/barebox.inc | 5 +++++
 1 file changed, 5 insertions(+)
meta/recipes-bsp/barebox/barebox.inc +5 -0
index a6c790b84b..4827f7b3a4 100644
@@ -48,6 +48,11 @@ EXTRA_OEMAKE = " \
 BAREBOX_CONFIG[doc] = "The barebox kconfig defconfig file. Not used if a file called defconfig is added to the SRC_URI."
 BAREBOX_CONFIG ?= ""
 
+# set defaults for oe-core MACHINEs to allow building barebox
+BAREBOX_CONFIG:qemuarm = "vexpress_defconfig"
+BAREBOX_CONFIG:qemuarm64 = "qemu_virt64_defconfig"
+BAREBOX_CONFIG:qemux86-64 = "efi_defconfig"
+
 do_configure() {
         if [ -e ${WORKDIR}/defconfig ]; then
                 cp ${WORKDIR}/defconfig ${B}/.config
Signature · 1 line
2.39.2
#4 lore raw

Re: [PATCH v2 3/9] barebox: set default BAREBOX_CONFIG for qemu machines

meta/recipes-bsp/barebox/barebox.inc EXTRA_OEMAKE = " \ · line 54

Quoted 4 lines · Enrico Jorns
meta/recipes-bsp/barebox/barebox.inc +4 -0
+# set defaults for oe-core MACHINEs to allow building barebox
+BAREBOX_CONFIG:qemuarm = "vexpress_defconfig"
+BAREBOX_CONFIG:qemuarm64 = "qemu_virt64_defconfig"
+BAREBOX_CONFIG:qemux86-64 = "efi_defconfig"

As all three are different perhaps there could be commentary in the recipe explaining why that and how they differ? Note that there's also a qemuall override if there's a single configuration that works for all qemu targets.

Alex

#5 lore raw

Re: [PATCH v2 3/9] barebox: set default BAREBOX_CONFIG for qemu machines

Hi Alex,

Am Freitag, dem 31.03.2023 um 16:04 +0200 schrieb Alexander Kanavin:

Quoted 10 lines · Enrico Jorns

meta/recipes-bsp/barebox/barebox.inc EXTRA_OEMAKE = " \ · line 54

Quoted 4 lines
meta/recipes-bsp/barebox/barebox.inc +4 -0
+# set defaults for oe-core MACHINEs to allow building barebox
+BAREBOX_CONFIG:qemuarm = "vexpress_defconfig"
+BAREBOX_CONFIG:qemuarm64 = "qemu_virt64_defconfig"
+BAREBOX_CONFIG:qemux86-64 = "efi_defconfig"

As all three are different perhaps there could be commentary in the recipe explaining why that and how they differ? Note that there's also a qemuall override if there's a single configuration that works for all qemu targets.

as these three are different archs/machines, I do not find it surprising that they require different configs. I assumed that this was self-explaining.

Note that for UBOOT, you have

in qemux86-64.conf:

UBOOT_MACHINE ?= "qemu-x86_64_defconfig"

in qemuarm.conf:

UBOOT_MACHINE ?= "qemu_arm_defconfig"

in qemuarm64.conf:

UBOOT_MACHINE ?= "qemu_arm64_defconfig"

which is pretty much the same as for barebox, isn't it? Only the naming scheme is a bit more consistent probably.

Thanks and regards

Enrico

Quoted 3 lines

Alex

Signature · 4 lines
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#6 lore raw

Re: [PATCH v2 3/9] barebox: set default BAREBOX_CONFIG for qemu machines

Quoted 10 lines · Enrico Jörns
Quoted 9 lines

meta/recipes-bsp/barebox/barebox.inc EXTRA_OEMAKE = " \ · line 54

Quoted 4 lines
meta/recipes-bsp/barebox/barebox.inc +4 -0
+# set defaults for oe-core MACHINEs to allow building barebox
+BAREBOX_CONFIG:qemuarm = "vexpress_defconfig"
+BAREBOX_CONFIG:qemuarm64 = "qemu_virt64_defconfig"
+BAREBOX_CONFIG:qemux86-64 = "efi_defconfig"

As all three are different perhaps there could be commentary in the recipe explaining why that and how they differ? Note that there's also a qemuall override if there's a single configuration that works for all qemu targets.

Quoted 2 lines

as these three are different archs/machines, I do not find it surprising that they require different configs. I assumed that this was self-explaining.

Not at all. What is this 'vexpress' thingy? What does 'express' mean? Why does it apply to arm, but not arm64? Why arm64 is using 'qemu_virt64', but not any other targets, if that config does not seem arm-specific? Why is x86 using 'efi' but not the arm targets?

The choices you made here should be explained.

Alex

#7 lore raw

Re: [PATCH v2 3/9] barebox: set default BAREBOX_CONFIG for qemu machines

Am Dienstag, dem 11.04.2023 um 16:16 +0200 schrieb Alexander Kanavin:

Quoted 15 lines · Enrico Jörns
Quoted 10 lines
Quoted 9 lines

meta/recipes-bsp/barebox/barebox.inc EXTRA_OEMAKE = " \ · line 54

Quoted 4 lines
meta/recipes-bsp/barebox/barebox.inc +4 -0
+# set defaults for oe-core MACHINEs to allow building barebox
+BAREBOX_CONFIG:qemuarm = "vexpress_defconfig"
+BAREBOX_CONFIG:qemuarm64 = "qemu_virt64_defconfig"
+BAREBOX_CONFIG:qemux86-64 = "efi_defconfig"

As all three are different perhaps there could be commentary in the recipe explaining why that and how they differ? Note that there's also a qemuall override if there's a single configuration that works for all qemu targets.

Quoted 3 lines

as these three are different archs/machines, I do not find it surprising that they require different
configs. I assumed that this was self-explaining.

Quoted 6 lines

Not at all. What is this 'vexpress' thingy? What does 'express' mean? Why does it apply to arm, but not arm64? Why arm64 is using 'qemu_virt64', but not any other targets, if that config does not seem arm-specific? Why is x86 using 'efi' but not the arm targets?

The choices you made here should be explained.

Vexpress is "Versatile Express" which once was the ARMv7 reference platform and is one of the platforms emulated in QEMU (when using "-machine vexpress-a9" or "-machine vexpress-a15").

virt, or virt64 are the generic QEMU ARM platform. For historical reasons, there is only a dedicated vexpress defconfig in barebox.

EFI is used for x86 only, since on ARM barebox is normally used as the primary loader only.

However, since barebox builds are multi-platform builds by default, it should be possible to use the standard multi_v7_defconfig and multi_v8_defconfig instead (for arm). I'll give it a try.

Regards, Enrico

Quoted 3 lines

Alex

Signature · 4 lines
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#8 lore raw

PATCH v2 4/9 oeqa/utils/qemurunner: support ignoring vt100 escape sequences

In reply to Enrico Jorns

If we talk to terminals that like colors, we need to ignore the vt100 escape sequences when matching strings.

---
 meta/lib/oeqa/utils/qemurunner.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/lib/oeqa/utils/qemurunner.py +2 -1
index 05385763ac..95c9e6596c 100644
@@ -30,6 +30,7 @@ control_range = list(range(0,32))+list(range(127,160))
 control_chars = [chr(x) for x in control_range
                 if chr(x) not in string.printable]
 re_control_char = re.compile('[%s]' % re.escape("".join(control_chars)))
+re_vt100 = re.compile(r'(\x1b\[|\x9b)[^@-_a-z]*[@-_a-z]|\x1b[@-_a-z]')
 
 class QemuRunner:
 
@@ -662,7 +663,7 @@ class QemuRunner:
                 time.sleep(0.1)
                 answer = self.server_socket.recv(1024)
                 if answer:
-                    data += answer.decode('utf-8')
+                    data += re_vt100.sub("", answer.decode('utf-8'))
                     # Search the prompt to stop
                     if re.search(self.boot_patterns['search_cmd_finished'], data):
                         break
Signature · 1 line
2.39.2
#9 lore raw

Re: [PATCH v2 4/9] oeqa/utils/qemurunner: support ignoring vt100 escape sequences

Quoted 1 line · Enrico Jorns

+re_vt100 = re.compile(r'(\x1b\[|\x9b)[^@-_a-z]*[@-_a-z]|\x1b[@-_a-z]')

This piece of 'magic' needs to be explained :) What does it define?

Alex

#10 lore raw

Re: [PATCH v2 4/9] oeqa/utils/qemurunner: support ignoring vt100 escape sequences

Hi Alex,

Am Freitag, dem 31.03.2023 um 16:05 +0200 schrieb Alexander Kanavin:

Quoted 4 lines · Enrico Jorns
Quoted 1 line

+re_vt100 = re.compile(r'(\x1b\[|\x9b)[^@-_a-z]*[@-_a-z]|\x1b[@-_a-z]')

This piece of 'magic' needs to be explained :) What does it define?

this is actually 'stolen' from labgrid's implementation for barebox[1] or shell[2] drivers which have exactly the same needs: They need to remove the ansii (color) control codes from the strings in order to match the text only.

The unprocessed barebox console prompt would look like:

ESC[1;32mbarebox@ESC[1;36mARM QEMU virt64:/ESC[0m

where we cannot match for something like "barebox@ARM QEMU virt64:/". The same applies to colored linux terminal output of course.

The "\x1b\[" from the regex catches the standard start of ansii escape sequence while the rest catches the actual command code executed.

Regards, Enrico

[1] https://github.com/labgrid-project/labgrid/blob/e05f530c22513cd775b4f84e28f6c8c920b95102/labgrid/driver/bareboxdriver.py#L44
[2] https://github.com/labgrid-project/labgrid/blob/5d3d5714976b37df0f9a9b769c581fe1f83ccbdc/labgrid/driver/shelldriver.py#L61

Quoted 2 lines

Alex

Signature · 4 lines
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#11 lore raw

Re: [PATCH v2 4/9] oeqa/utils/qemurunner: support ignoring vt100 escape sequences

Right, but this information should be recorded in commit message, or the file itself.

Alex

Quoted 38 lines · Enrico Jörns

Hi Alex,

Am Freitag, dem 31.03.2023 um 16:05 +0200 schrieb Alexander Kanavin:

Quoted 4 lines · Enrico Jorns
Quoted 1 line

+re_vt100 = re.compile(r'(\x1b\[|\x9b)[^@-_a-z]*[@-_a-z]|\x1b[@-_a-z]')

This piece of 'magic' needs to be explained :) What does it define?

this is actually 'stolen' from labgrid's implementation for barebox[1] or shell[2] drivers which have exactly the same needs: They need to remove the ansii (color) control codes from the strings in order to match the text only.

The unprocessed barebox console prompt would look like:

ESC[1;32mbarebox@ESC[1;36mARM QEMU virt64:/ESC[0m

where we cannot match for something like "barebox@ARM QEMU virt64:/". The same applies to colored linux terminal output of course.

The "\x1b\[" from the regex catches the standard start of ansii escape sequence while the rest catches the actual command code executed.

Regards, Enrico

[1] https://github.com/labgrid-project/labgrid/blob/e05f530c22513cd775b4f84e28f6c8c920b95102/labgrid/driver/bareboxdriver.py#L44
[2] https://github.com/labgrid-project/labgrid/blob/5d3d5714976b37df0f9a9b769c581fe1f83ccbdc/labgrid/driver/shelldriver.py#L61

Quoted 2 lines

Alex

--

Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#12 lore raw

PATCH v2 5/9 oeqa/utils/qemurunner: simplify output parsing and make crlf-compatible

In reply to Enrico Jorns

Reduces the nesting depth and replaces the manual newline matching by built-in splitlines() method.

This makes it compatible with shells that use windows-compatible line breaks, e.g. for EFI loaders.

More comments and an early return handling should make the code a bit more readable.

---
 meta/lib/oeqa/utils/qemurunner.py | 32 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 17 deletions(-)
meta/lib/oeqa/utils/qemurunner.py +15 -17
index 95c9e6596c..1e544acc90 100644
@@ -672,23 +672,21 @@ class QemuRunner:
                         return (1, "")
                     raise Exception("No data on serial console socket, connection closed?")
 
-        if data:
-            if raw:
-                status = 1
-            else:
-                # Remove first line (command line) and last line (prompt)
-                data = data[data.find('$?\r\n')+4:data.rfind('\r\n')]
-                index = data.rfind('\r\n')
-                if index == -1:
-                    status_cmd = data
-                    data = ""
-                else:
-                    status_cmd = data[index+2:]
-                    data = data[:index]
-                if (status_cmd == "0"):
-                    status = 1
-        return (status, str(data))
-
+        # If we got no data, we assume something went wrong and return 0
+        if not data:
+            return (0, str(None))
+
+        # in raw mode, we cannot check exit status output and thus assume success
+        if raw:
+            return (1, str(data))
+
+        # Split lines into array and remove first line (command line) and last line (prompt)
+        # Also remove empty lines to ease catching results
+        outlines = list(filter(None, data.splitlines()[1:-1]))
+        # Remaining last line contains exit code output
+        if (outlines[-1] == "0"):
+            status = 1
+        return (status, "\n".join(outlines[0:-1]))
 
     def _dump_host(self):
         self.host_dumper.create_dir("qemu")
Signature · 1 line
2.39.2
#13 lore raw

PATCH v2 6/9 oeqa/utils/commands: document runqemu context manager

In reply to Enrico Jorns

Usage and arguments are not necessarily self-explaining.

---
 meta/lib/oeqa/utils/commands.py | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
meta/lib/oeqa/utils/commands.py +16 -1
index f4daea2507..efbb6ab6f2 100644
@@ -314,7 +314,22 @@ def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec=
 @contextlib.contextmanager
 def runqemu(pn, ssh=True, runqemuparams='', image_fstype=None, launch_cmd=None, qemuparams=None, overrides={}, discard_writes=True):
     """
-    launch_cmd means directly run the command, don't need set rootfs or env vars.
+    Starts a context manager for a 'oeqa.targetcontrol.QemuTarget' resource.
+    The underlying Qemu will be booted into a shell when the generator yields.
+
+    Usage:
+
+        with runqemu('core-image-minimal') as qemu:
+            qemu.run_serial('cat /proc/cpuinfo')
+
+    Args:
+        pn (str): (image) recipe to run on
+        ssh (boolean): whether or not to enable SSH (network access)
+        runqemuparams (str): space-separated list of params to pass to 'runqemu' script (like 'nographics', 'ovmf', etc.)
+        image_fstype (str): IMAGE_FSTYPE to use
+        launch_cmd (str): directly run this command and bypass automatic runqemu parameter generation
+        overrides (dict): dict of "'<bitbake-variable>': value" pairs that allows overriding bitbake variables
+        discard_writes (boolean): enables qemu -snapshot feature to prevent modifying original image
     """
 
     import bb.tinfoil
Signature · 1 line
2.39.2
#14 lore raw

PATCH v2 7/9 oeqa: support passing custom boot patterns to runqemu

In reply to Enrico Jorns

This allows defining non-standard patterns from QEMU tests just as they are already supported by the testimage.bbclass.

Will allow testing non-Linux shells in QEMU, too, e.g. a U-Boot or other bootloader shell.

---
 meta/lib/oeqa/targetcontrol.py    | 5 +++--
 meta/lib/oeqa/utils/commands.py   | 5 +++--
 meta/lib/oeqa/utils/qemurunner.py | 2 +-
 3 files changed, 7 insertions(+), 5 deletions(-)
meta/lib/oeqa/targetcontrol.py +3 -2
index 1fdff82889..43a213ac0e 100644
@@ -92,7 +92,7 @@ class QemuTarget(BaseTarget):
 
     supported_image_fstypes = ['ext3', 'ext4', 'cpio.gz', 'wic']
 
-    def __init__(self, d, logger, image_fstype=None):
+    def __init__(self, d, logger, image_fstype=None, boot_patterns=None):
 
         import oe.types
 
@@ -148,7 +148,8 @@ class QemuTarget(BaseTarget):
                             dump_host_cmds = d.getVar("testimage_dump_host"),
                             logger = logger,
                             tmpfsdir = d.getVar("RUNQEMU_TMPFS_DIR"),
-                            serial_ports = len(d.getVar("SERIAL_CONSOLES").split()))
+                            serial_ports = len(d.getVar("SERIAL_CONSOLES").split()),
+                            boot_patterns = boot_patterns)
 
         self.target_dumper = TargetDumper(dump_target_cmds, dump_dir, self.runner)
         self.monitor_dumper = MonitorDumper(dump_monitor_cmds, dump_dir, self.runner)
meta/lib/oeqa/utils/commands.py +3 -2
index efbb6ab6f2..1f6a617367 100644
@@ -312,7 +312,7 @@ def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec=
         f.write('LAYERSERIES_COMPAT_%s = "%s"\n' % (templayername, corenames))
 
 @contextlib.contextmanager
-def runqemu(pn, ssh=True, runqemuparams='', image_fstype=None, launch_cmd=None, qemuparams=None, overrides={}, discard_writes=True):
+def runqemu(pn, ssh=True, runqemuparams='', image_fstype=None, launch_cmd=None, qemuparams=None, overrides={}, boot_patterns = {}, discard_writes=True):
     """
     Starts a context manager for a 'oeqa.targetcontrol.QemuTarget' resource.
     The underlying Qemu will be booted into a shell when the generator yields.
@@ -329,6 +329,7 @@ def runqemu(pn, ssh=True, runqemuparams='', image_fstype=None, launch_cmd=None,
         image_fstype (str): IMAGE_FSTYPE to use
         launch_cmd (str): directly run this command and bypass automatic runqemu parameter generation
         overrides (dict): dict of "'<bitbake-variable>': value" pairs that allows overriding bitbake variables
+        boot_patterns (dict): dict of "'<pattern-name>': value" pairs to override default boot patterns, e.g. when not booting Linux
         discard_writes (boolean): enables qemu -snapshot feature to prevent modifying original image
     """
 
@@ -360,7 +361,7 @@ def runqemu(pn, ssh=True, runqemuparams='', image_fstype=None, launch_cmd=None,
 
         logdir = recipedata.getVar("TEST_LOG_DIR")
 
-        qemu = oeqa.targetcontrol.QemuTarget(recipedata, targetlogger, image_fstype)
+        qemu = oeqa.targetcontrol.QemuTarget(recipedata, targetlogger, image_fstype, boot_patterns=boot_patterns)
     finally:
         # We need to shut down tinfoil early here in case we actually want
         # to run tinfoil-using utilities with the running QEMU instance.
meta/lib/oeqa/utils/qemurunner.py +1 -1
index 1e544acc90..0d64df7253 100644
@@ -93,7 +93,7 @@ class QemuRunner:
 
         # Only override patterns that were set e.g. login user TESTIMAGE_BOOT_PATTERNS[send_login_user] = "webserver\n"
         for pattern in accepted_patterns:
-            if not self.boot_patterns[pattern]:
+            if pattern not in self.boot_patterns or not self.boot_patterns[pattern]:
                 self.boot_patterns[pattern] = default_boot_patterns[pattern]
 
     def create_socket(self):
Signature · 1 line
2.39.2
#15 lore raw

PATCH v2 8/9 oeqa/selftest/cases: add barebox tests

In reply to Enrico Jorns
---
 meta/lib/oeqa/selftest/cases/barebox.py | 72 +++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/barebox.py
meta/lib/oeqa/selftest/cases/barebox.py new file +72 -0
new file mode 100644
index 0000000000..497cd6e8dd
@@ -0,0 +1,72 @@
+# Qemu-based barebox bootloader integration testing
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, runqemu
+from oeqa.core.decorator.data import skipIfNotArch
+from oeqa.core.decorator import OETestTag
+
+barebox_boot_patterns = {
+        'search_reached_prompt': r"stop autoboot",
+        'search_login_succeeded': r"barebox@[^:]+:[^ ]+ ",
+        'search_cmd_finished': r"barebox@[a-zA-Z0-9\-\s]+:/"
+        }
+
+
+class BareboxTest(OESelftestTestCase):
+
+    @skipIfNotArch(['arm', 'aarch64'])
+    @OETestTag("runqemu")
+    def test_boot_barebox(self):
+        """
+        Tests building barebox and booting it with QEMU
+        """
+
+        self.write_config("""
+QB_DEFAULT_KERNEL = "barebox-dt-2nd.img"
+PREFERRED_PROVIDER_virtual/bootloader = "barebox"
+""")
+
+        bitbake("virtual/bootloader core-image-minimal")
+
+        with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic',
+                     boot_patterns=barebox_boot_patterns) as qemu:
+
+            # test if barebox console works
+            cmd = "version"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(status, 1, msg=output)
+            self.assertTrue("barebox" in output, msg=output)
+
+    @skipIfNotArch(['x86_64'])
+    @OETestTag("runqemu")
+    def test_boot_barebox_efi(self):
+        """
+        Tests building barebox for UEFI and booting it as EFI payload
+        with QEMU + OVMF
+        """
+        image = "core-image-minimal"
+
+        self.write_config("""
+IMAGE_INSTALL:append = " barebox"
+MACHINE_FEATURES:append = " pcbios efi"
+EXTRA_IMAGEDEPENDS += "ovmf"
+EFI_PROVIDER = "barebox"
+IMAGE_FSTYPES += "wic"
+WKS_FILE = "efi-bootdisk.wks.in"
+""")
+
+        bitbake(image)
+
+        with runqemu(image, ssh=False, runqemuparams='nographic ovmf',
+                     boot_patterns=barebox_boot_patterns, image_fstype='wic') as qemu:
+
+            # test if barebox console works
+            cmd = "version"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(status, 1, msg=output)
+            self.assertTrue("barebox" in output, msg=output)
Signature · 1 line
2.39.2
#16 lore raw

PATCH v2 9/9 oeqa/selftest/cases: add basic u-boot test

In reply to Enrico Jorns
---
 meta/lib/oeqa/selftest/cases/uboot.py | 40 +++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/uboot.py
meta/lib/oeqa/selftest/cases/uboot.py new file +40 -0
new file mode 100644
index 0000000000..d3c60c3052
@@ -0,0 +1,40 @@
+# Qemu-based u-boot bootloader integration testing
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, runqemu
+from oeqa.core.decorator import OETestTag
+
+uboot_boot_patterns = {
+        'search_reached_prompt': "stop autoboot",
+        'search_login_succeeded': "=>",
+        'search_cmd_finished': "=>"
+        }
+
+
+class UBootTest(OESelftestTestCase):
+
+    @OETestTag("runqemu")
+    def test_boot_uboot(self):
+        """
+        Tests building u-boot and booting it with QEMU
+        """
+
+        self.write_config("""
+QB_DEFAULT_BIOS = "u-boot.bin"
+PREFERRED_PROVIDER_virtual/bootloader = "u-boot"
+""")
+        bitbake("virtual/bootloader core-image-minimal")
+
+        with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic',
+                     boot_patterns=uboot_boot_patterns) as qemu:
+
+            # test if u-boot console works
+            cmd = "version"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(status, 1, msg=output)
+            self.assertTrue("U-Boot" in output, msg=output)
Signature · 1 line
2.39.2
#17 lore raw
In reply to Enrico Jorns
Quoted 8 lines · Enrico Jorns

v2:
* set myself as maintainer in maintainers.inc
* move doc from documentation.conf to recipe
* update to barebox v2023.03 (including fixes for musl, etc.)
* set standard configs for qemu machines to allow building them
* add better efi dir support in barebox recipe
* enable testing bootloaders in oeqa
* add test cases for barebox (and u-boot)

Thanks for persevering with this! The patchset is mostly fine (I'll write a couple comments now), and just needs to get a verdict from the autobuilder.

Alex

#18 lore raw
In reply to Enrico Jorns
Quoted 24 lines · Enrico Jorns

This adds support for the barebox bootloader (and tools) to oe-core.

In order to have proper testing, this extends oe-selftest to allow basic testing of bootloaders. While at it, cover both barebox and u-boot.

v2:
* set myself as maintainer in maintainers.inc
* move doc from documentation.conf to recipe
* update to barebox v2023.03 (including fixes for musl, etc.)
* set standard configs for qemu machines to allow building them
* add better efi dir support in barebox recipe
* enable testing bootloaders in oeqa
* add test cases for barebox (and u-boot)

Enrico Jorns (7):
barebox: set default BAREBOX_CONFIG for qemu machines oeqa/utils/qemurunner: support ignoring vt100 escape sequences oeqa/utils/qemurunner: simplify output parsing and make
crlf-compatible
oeqa/utils/commands: document runqemu context manager
oeqa: support passing custom boot patterns to runqemu oeqa/selftest/cases: add barebox tests oeqa/selftest/cases: add basic u-boot test

Quoted 3 lines

Marco Felsch (2):
barebox: add initial support
barebox-tools: add initial barebox tools support

Testing on the autobuilder did trigger a few issues:

on musl:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5023
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/6932

and some failing oe-selftests:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/4971
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5023
[there were more but look similar]

Cheers,

Richard

#19 lore raw

Hi Richard,

Am Montag, dem 03.04.2023 um 16:49 +0100 schrieb Richard Purdie:

Quoted 33 lines · Enrico Jorns
Quoted 24 lines

This adds support for the barebox bootloader (and tools) to oe-core.

In order to have proper testing, this extends oe-selftest to allow basic testing of bootloaders. While at it, cover both barebox and u-boot.

v2:
  * set myself as maintainer in maintainers.inc
  * move doc from documentation.conf to recipe
  * update to barebox v2023.03 (including fixes for musl, etc.)
  * set standard configs for qemu machines to allow building them
  * add better efi dir support in barebox recipe
  * enable testing bootloaders in oeqa
  * add test cases for barebox (and u-boot)

Enrico Jorns (7):
  barebox: set default BAREBOX_CONFIG for qemu machines oeqa/utils/qemurunner: support ignoring vt100 escape sequences oeqa/utils/qemurunner: simplify output parsing and make crlf-compatible
  oeqa/utils/commands: document runqemu context manager
  oeqa: support passing custom boot patterns to runqemu oeqa/selftest/cases: add barebox tests oeqa/selftest/cases: add basic u-boot test

Quoted 3 lines

Marco Felsch (2):
  barebox: add initial support
  barebox-tools: add initial barebox tools support

Testing on the autobuilder did trigger a few issues:

on musl:

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5023
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/6932

and some failing oe-selftests:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/4971
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5023
[there were more but look similar]

thank you for the autobuilder logs!

I fear we are running into a conceptional issue that I already feared we could run into (but did not test for yet).

If I traced this down correctly, then the 'issue' is:

| NOTE: Multiple providers are available for virtual/bootloader (barebox, u-boot) | Consider defining a PREFERRED_PROVIDER entry to match virtual/bootloader

The fitimage tests building "virtual/bootloader", silently assuming this is equal to u-boot (which it actually was so far).

Now, the quick fix for this would be to properly set the correct PREFERRED_PROVIDER here, but I could imagine that we run into similar issues in other builds, too.

Do you have a good solution for this at hand? Like defining a standard preferred provider for virtual/bootloader?

I guess one could argue possibly that the mechanism does exactly what it is meant for. And a few machines (like beaglebone-yocto or qemuloongarch) actually set the PREFERRED_PROVIDER, but not all and this might lead to silently switching the bootloader (like in our tests).

I hope you have a better overview on the topic than I have and could give me a hint on how you would like to see this being resolved?

Thanks in advance and best regards

Enrico

Quoted 5 lines

Cheers,

Richard

Signature · 4 lines
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#20 lore raw
Quoted 65 lines · Enrico Jörns

Hi Richard,

Am Montag, dem 03.04.2023 um 16:49 +0100 schrieb Richard Purdie:

Quoted 33 lines · Enrico Jorns
Quoted 24 lines

This adds support for the barebox bootloader (and tools) to oe-core.

In order to have proper testing, this extends oe-selftest to allow basic testing of bootloaders. While at it, cover both barebox and u-boot.

v2:
* set myself as maintainer in maintainers.inc
* move doc from documentation.conf to recipe
* update to barebox v2023.03 (including fixes for musl, etc.)
* set standard configs for qemu machines to allow building them
* add better efi dir support in barebox recipe
* enable testing bootloaders in oeqa
* add test cases for barebox (and u-boot)

Enrico Jorns (7):
barebox: set default BAREBOX_CONFIG for qemu machines oeqa/utils/qemurunner: support ignoring vt100 escape sequences oeqa/utils/qemurunner: simplify output parsing and make
crlf-compatible
oeqa/utils/commands: document runqemu context manager
oeqa: support passing custom boot patterns to runqemu oeqa/selftest/cases: add barebox tests oeqa/selftest/cases: add basic u-boot test

Quoted 3 lines

Marco Felsch (2):
barebox: add initial support
barebox-tools: add initial barebox tools support

Testing on the autobuilder did trigger a few issues:

on musl:

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5023
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/6932

and some failing oe-selftests:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/4971
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5023
[there were more but look similar]

thank you for the autobuilder logs!

I fear we are running into a conceptional issue that I already feared we could run into (but did not test for yet).

If I traced this down correctly, then the 'issue' is:

| NOTE: Multiple providers are available for virtual/bootloader (barebox, u-boot) | Consider defining a PREFERRED_PROVIDER entry to match virtual/bootloader

The fitimage tests building "virtual/bootloader", silently assuming this is equal to u-boot (which it actually was so far).

Now, the quick fix for this would be to properly set the correct PREFERRED_PROVIDER here, but I could imagine that we run into similar issues in other builds, too.

Do you have a good solution for this at hand? Like defining a standard preferred provider for virtual/bootloader?

Quoted 6 lines

I guess one could argue possibly that the mechanism does exactly what it is meant for. And a few machines (like beaglebone-yocto or qemuloongarch) actually set the PREFERRED_PROVIDER, but not all and this might lead to silently switching the bootloader (like in our tests).

I hope you have a better overview on the topic than I have and could give me a hint on how you would like to see this being resolved?

I see that it fails to build with clang
https://errors.yoctoproject.org/Errors/Details/699467/

is it buildable with clang ? if not we can pin it to use gcc always in meta-clang but I want to make sure first.

Quoted 27 lines

Thanks in advance and best regards

Enrico

Quoted 5 lines

Cheers,

Richard

--

Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#179655): https://lists.openembedded.org/g/openembedded-core/message/179655
Mute This Topic: https://lists.openembedded.org/mt/97970644/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#21 lore raw

Hi Khem,

Quoted 7 lines

Am Montag, dem 03.04.2023 um 19:33 -0700 schrieb Khem Raj:

[...]

I see that it fails to build with clang
https://errors.yoctoproject.org/Errors/Details/699467/

thanks for testing this!

Quoted 2 lines

is it buildable with clang ? if not we can pin it to use gcc always in meta-clang but I want to make sure first.

barebox itself is unfortunately not buildable with clang since on some architectures like ARM32 it uses GCC extensions that are not implemented by clang.

The barebox-tools however that are built for 'sandbox' should be buildable with clang. It's just that when passing CROSS_COMPILE, barebox cannot guess if it has to append 'clang' or 'gcc' suffix. The kernel seems to have an explicit handling for this by checking for "ifneq ($(LLVM),)" and plugging together CC differently.

With barebox we could however still override the Makefile's default "CC = $(CROSS_COMPILE)gcc" when passing CC explicitly to EXTRA_OEMAKE. We have just successfully tested this patch:

meta/recipes-bsp/barebox/barebox-tools_2023.03.0.bb +2 -0
@@ -18,6 +18,8 @@ EXTRA_OEMAKE = " \
   ARCH=sandbox \
   CROSS_COMPILE=${TARGET_PREFIX} -C ${S} O=${B} \
   CROSS_PKG_CONFIG=pkg-config \
+  CC='${CC}' \
+  LD='${LD}' \
   "
 
 

If that looks fine for you, I could add it to the recipe.

Best regards, Enrico

Quoted 29 lines
Quoted 28 lines

Thanks in advance and best regards

Enrico

Quoted 5 lines

Cheers,

Richard

--
Pengutronix e.K.                           | Enrico Jörns                | Embedded Linux Consulting & Support        | https://www.pengutronix.de/ | Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  | 31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#179655): https://lists.openembedded.org/g/openembedded-core/message/179655
Mute This Topic: https://lists.openembedded.org/mt/97970644/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 4 lines
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#22 lore raw
Quoted 38 lines · Enrico Jörns

Hi Khem,

Quoted 7 lines

Am Montag, dem 03.04.2023 um 19:33 -0700 schrieb Khem Raj:

[...]

I see that it fails to build with clang
https://errors.yoctoproject.org/Errors/Details/699467/

thanks for testing this!

Quoted 2 lines

is it buildable with clang ? if not we can pin it to use gcc always in meta-clang but I want to make sure first.

barebox itself is unfortunately not buildable with clang since on some architectures like ARM32 it uses GCC extensions that are not implemented by clang.

The barebox-tools however that are built for 'sandbox' should be buildable with clang. It's just that when passing CROSS_COMPILE, barebox cannot guess if it has to append 'clang' or 'gcc' suffix. The kernel seems to have an explicit handling for this by checking for "ifneq ($(LLVM),)" and plugging together CC differently.

With barebox we could however still override the Makefile's default "CC = $(CROSS_COMPILE)gcc" when passing CC explicitly to EXTRA_OEMAKE. We have just successfully tested this patch:

meta/recipes-bsp/barebox/barebox-tools_2023.03.0.bb +2 -0
@@ -18,6 +18,8 @@ EXTRA_OEMAKE = " \
   ARCH=sandbox \
   CROSS_COMPILE=${TARGET_PREFIX} -C ${S} O=${B} \
   CROSS_PKG_CONFIG=pkg-config \
+  CC='${CC}' \
+  LD='${LD}' \
   "
 
 
Quoted 2 lines

If that looks fine for you, I could add it to the recipe.

yeah I was testing one with CC alone and that worked too but also containing LD is good, so please include above
changes in next patchset

Quoted 39 lines

Best regards, Enrico

Quoted 20 lines
Quoted 20 lines

Thanks in advance and best regards

Enrico

Quoted 5 lines

Cheers,

Richard

--

Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
Quoted 9 lines
Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#179655): https://lists.openembedded.org/g/openembedded-core/message/179655
Mute This Topic: https://lists.openembedded.org/mt/97970644/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--

Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |