Runtime testing infrastructure

Hello,

Here is a third version of a small runtime testing infrastructure for Buildroot.

I have not tried to take into account all suggestions from the review, only the ones that really must be fixed before it gets applied. I'd like this stuff to be applied so that more people can start working on the testing infrastructure and submit patches.

The first patch introduces the infrastructure itself, the other 4 patches just add a small number of test cases.

By far and large the most featureful set of tests are the filesystem tests, testing a lot of features of our filesystem image support.

There are definitely lots of possible improvements to the infrastructure, and lots of possible tests to add, but we need to get started at some point, and that's an initial proposal.

Here is a quick start:

1. List the tests

./support/testing/run-tests -l

2. Run a test:

./support/testing/run-tests -d <download dir> -o <output dir> <name of test>

e.g:

./support/testing/run-tests -d ~/dl/ -o ../outputs/ tests.fs.test_ubi.TestUbi

This patch series is also available at:

http://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=runtime-tests

Changes since v2:

- Replace binary files in the test rootfs overlay by text files, suggested by Yann.

- Use CSV files for the post-{build,image} tests output files, as suggested by Yann.

- Fix a bug in the toolchain tests, where the base class TestExternalToolchain was seen as a test case, even though it wasn't. To fix this, as suggested by Ricardo, rename the method from test_run() to common_check().

Changes since v1:

- Improvement to the external toolchain test cases:

* We are now checking that there are no broken links in target/ and staging/

* We are now verifying that the program interpreter of the busybox binary really exists on the target

* Some code factorized in a TestExternalToolchain() base class

* We are now boot testing the ARMv7/Thumb2 system generated with the Sourcery toolchain

* Additional test case for the Linaro ARM toolchain

- Fix i386 builtin kernel usage by the Emulator() class. Noticed by Ricardo.

- Fix the shebang of run-tests to use python2 explicitly, since we're python2 only for the moment. Noticed by Ricardo.

- Remove useless "import string" from run-tests. Noticed by Ricardo.

- Make BRTest.downloaddir and BRTest.outputdir absolute, to avoid issues. It fixes a bug with one particular test case. Noticed by Ricardo.

- Fix artefacts -> artifacts, suggested by Luca.

- Add docstring for the smart_open() and get_elf_arch_tag() helpers, suggested by Luca.

- Fix self.logtofile test in builder.py, noticed by Luca.

- Use 'cf' as config file handle variable, rather than 'cfd', suggested by Luca.

- Improve the Emulator.boot() method documentation about builtin kernel, suggested by Luca.

- Remove excessive much logging from the Emulator() class.

- Run "dmesg -n 1" as the first command once logged in, to avoid kernel messages polluting the console parsing. Issue reported by Ricardo.

- Remove Emulator.showlog() method, not used. Noticed by Luca.

- Use BR2_DL_DIR when option -d is not passed. Suggested by Luca.

- Create output directory if it doesn't exist. Suggested by Luca.

- Print help of run-tests when there is a failure.

- Remove bogus TODO, noticed by Ricardo.

- Fix some of the ISO9660 test cases, that were wrongly expecting a read-only filesystem, while they really get a read/write filesystem. Noticed by Ricardo.

- Use "testpwd" as the root password for the Dropbear test case instead of "root", which could be confused with the login name.

- Don't quiet grep commands in test cases, as their output can be useful. Suggested by Ricardo.

- Use Emulator.login() instead of Emulator.login("root") to simply login as root with no password. Indeed, the argument of the login() method is the password, not the login name. Noticed by Ricardo.

Best regards,

Thomas Petazzoni

Thomas Petazzoni (5):
support/testing: core testing infrastructure
support/testing: add core tests
support/testing: add fs tests
support/testing: add package tests
support/testing: add toolchain tests

 support/testing/conf/grub-menu.lst                 |  20 +++
 support/testing/conf/grub2.cfg                     |   7 +
 support/testing/conf/isolinux.cfg                  |   5 +
 .../testing/conf/minimal-x86-qemu-kernel.config    |  23 +++
 support/testing/conf/unittest.cfg                  |   6 +
 support/testing/infra/__init__.py                  |  89 +++++++++++
 support/testing/infra/basetest.py                  |  66 +++++++++
 support/testing/infra/builder.py                   |  49 +++++++
 support/testing/infra/emulator.py                  | 135 +++++++++++++++++
 support/testing/run-tests                          |  83 +++++++++++
 support/testing/tests/__init__.py                  |   0
 support/testing/tests/core/__init__.py             |   0
 support/testing/tests/core/post-build.sh           |  12 ++
 support/testing/tests/core/post-image.sh           |  12 ++
 .../testing/tests/core/rootfs-overlay1/test-file1  |   1 +
 .../tests/core/rootfs-overlay2/etc/test-file2      |   1 +
 support/testing/tests/core/test_post_scripts.py    |  40 +++++
 support/testing/tests/core/test_rootfs_overlay.py  |  27 ++++
 support/testing/tests/core/test_timezone.py        |  66 +++++++++
 support/testing/tests/fs/__init__.py               |   0
 support/testing/tests/fs/test_ext.py               | 119 +++++++++++++++
 support/testing/tests/fs/test_iso9660.py           | 162 +++++++++++++++++++++
 support/testing/tests/fs/test_jffs2.py             |  45 ++++++
 support/testing/tests/fs/test_squashfs.py          |  37 +++++
 support/testing/tests/fs/test_ubi.py               |  39 +++++
 support/testing/tests/fs/test_yaffs2.py            |  12 ++
 support/testing/tests/package/__init__.py          |   0
 support/testing/tests/package/test_dropbear.py     |  28 ++++
 support/testing/tests/package/test_python.py       |  35 +++++
 support/testing/tests/toolchain/__init__.py        |   0
 support/testing/tests/toolchain/test_external.py   | 156 ++++++++++++++++++++
 31 files changed, 1275 insertions(+)
 create mode 100644 support/testing/conf/grub-menu.lst
 create mode 100644 support/testing/conf/grub2.cfg
 create mode 100644 support/testing/conf/isolinux.cfg
 create mode 100644 support/testing/conf/minimal-x86-qemu-kernel.config
 create mode 100644 support/testing/conf/unittest.cfg
 create mode 100644 support/testing/infra/__init__.py
 create mode 100644 support/testing/infra/basetest.py
 create mode 100644 support/testing/infra/builder.py
 create mode 100644 support/testing/infra/emulator.py
 create mode 100755 support/testing/run-tests
 create mode 100644 support/testing/tests/__init__.py
 create mode 100644 support/testing/tests/core/__init__.py
 create mode 100755 support/testing/tests/core/post-build.sh
 create mode 100755 support/testing/tests/core/post-image.sh
 create mode 100644 support/testing/tests/core/rootfs-overlay1/test-file1
 create mode 100644 support/testing/tests/core/rootfs-overlay2/etc/test-file2
 create mode 100644 support/testing/tests/core/test_post_scripts.py
 create mode 100644 support/testing/tests/core/test_rootfs_overlay.py
 create mode 100644 support/testing/tests/core/test_timezone.py
 create mode 100644 support/testing/tests/fs/__init__.py
 create mode 100644 support/testing/tests/fs/test_ext.py
 create mode 100644 support/testing/tests/fs/test_iso9660.py
 create mode 100644 support/testing/tests/fs/test_jffs2.py
 create mode 100644 support/testing/tests/fs/test_squashfs.py
 create mode 100644 support/testing/tests/fs/test_ubi.py
 create mode 100644 support/testing/tests/fs/test_yaffs2.py
 create mode 100644 support/testing/tests/package/__init__.py
 create mode 100644 support/testing/tests/package/test_dropbear.py
 create mode 100644 support/testing/tests/package/test_python.py
 create mode 100644 support/testing/tests/toolchain/__init__.py
 create mode 100644 support/testing/tests/toolchain/test_external.py

2.7.4

This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation.

This core infrastructure commit includes the following aspects:

- A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases.

Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory).

- A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case.

- An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system.

- A run-tests script, which is the entry point to start the tests.

Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code!

---
 support/testing/conf/unittest.cfg |   6 ++
 support/testing/infra/__init__.py |  89 +++++++++++++++++++++++++
 support/testing/infra/basetest.py |  66 +++++++++++++++++++
 support/testing/infra/builder.py  |  49 ++++++++++++++
 support/testing/infra/emulator.py | 135 ++++++++++++++++++++++++++++++++++++++
 support/testing/run-tests         |  83 +++++++++++++++++++++++
 support/testing/tests/__init__.py |   0
 7 files changed, 428 insertions(+)
 create mode 100644 support/testing/conf/unittest.cfg
 create mode 100644 support/testing/infra/__init__.py
 create mode 100644 support/testing/infra/basetest.py
 create mode 100644 support/testing/infra/builder.py
 create mode 100644 support/testing/infra/emulator.py
 create mode 100755 support/testing/run-tests
 create mode 100644 support/testing/tests/__init__.py

new file mode 100644
index 0000000..6eaa234
@@ -0,0 +1,6 @@
+[unittest]
+plugins = nose2.plugins.mp
+
+[multiprocess]
+processes = 1
+always-on = True

new file mode 100644
index 0000000..e69de29

2.7.4

[..]

Thomas,

There are still 2 things that I think can be solved together.

On Mon, Mar 20, 2017 at 05:36 PM, Thomas Petazzoni wrote:

[snip]

[snip]

    self.__read_until("buildroot login:", 10)
  File "/tmp/testing-v3/support/testing/infra/emulator.py", line 89, in __read_until
    data = self.__tn.read_until(waitstr, timeout)
  File "/usr/lib/python2.7/telnetlib.py", line 294, in read_until
    return self._read_until_with_poll(match, timeout)
  File "/usr/lib/python2.7/telnetlib.py", line 329, in _read_until_with_poll
    self.fill_rawq()
  File "/usr/lib/python2.7/telnetlib.py", line 576, in fill_rawq
    buf = self.sock.recv(50)

         qemu_cmd = ["qemu-system-{}".format(qemu_arch),
-                    "-serial", "telnet::1234,server",
                     "-display", "none"]
@@ -75,4 +78,15 @@ class Emulator(object):
         with infra.smart_open(self.log_file) as lfh:
-            lfh.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
-            self.qemu = subprocess.Popen(qemu_cmd, stdout=lfh, stderr=lfh)
+            for port in range(TELNET_PORT_INITIAL, TELNET_PORT_LAST):
+                telnet_arg = ["-serial", "telnet::{},server".format(port)]
+                cmd = qemu_cmd + telnet_arg
+                lfh.write("> starting qemu with '{}'\n".format(" ".join(cmd)))
+                self.qemu = pexpect.spawn(cmd[0], cmd[1:], logfile=lfh)
+                ret = self.qemu.expect(["waiting for connection",

                 if self.__tn:
@@ -100,6 +114,8 @@ class Emulator(object):
     def login(self, password=None):
-        self.__read_until("buildroot login:", 10)
+        # The login prompt can take some time to appear when running multiple
+        # instances in parallel, so set the timeout to a large value
+        self.__read_until("buildroot login:", 30)

-        self.qemu.terminate()
-        self.qemu.kill()
+        self.qemu.terminate(force=True)

Hello,

Yes, this is a known problem, and there's a TODO about this in emulator.py:

# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss

# with the qemu machine.

Hum, do you know why this one happens? I definitely understand that will a qemu is running, another qemu cannot bind its telnet server on the same TCP port, but I don't see why it would cause a "Connection reset by peer".

Or is the logic of the test bogus, and if it doesn't manage to start qemu, the test still continues, and it in fact connects to an already running qemu instance from a different test, which then gets killed in the middle ?

It would probably be nicer to interact through stdio and pexpect, but in the mean time, this does the job. I'll integrate that into my series. Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Hello,

It is not clear to me why this one happens.

I redirected the output from Popen to stdout and stderr and when the error occurs only the messages from the first qemu appear before the Traceback:

[output/TestPythonBase/2017-03-23 20:52:29] Starting QEMU waiting for connection on: disconnected:telnet::1234,server
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument [output/TestPythonBase/2017-03-23 20:52:37] Cleaning up .[output/TestDropbear/2017-03-23 20:52:37] Starting [output/TestDropbear/2017-03-23 20:52:37] Cleaning up E

So I suppose it could be a bogus logic.

Anyway I think waiting for "waiting for connection" before connecting should be more reliable to know the qemu started than just trying to connect using telnet.

Regards,
Ricardo

Hi,

only a minor nit that I noticed while reviewing patch 5.

On 20/03/2017 21:36, Thomas Petazzoni wrote:
[...]

[...]

                     'bin/busybox')

/lib/ld-uClibc.so.0

"""

Luca

Hi,

this the review I had written a few days ago. Since the patch has just been applied, I've reworded it as a list of proposed improvements and request for clarifications.

Good we have this on master now!

Things that I already commented about and for which I plan to send patches:
- remove smart_open
- document code
- better reporting

[...]

  $ ./support/testing/run-tests
  [...]
  optional arguments:
    -h, --help            show this help message and exit
    --list, -l            list of available test cases

Luca

Hello,

Great!

Correct, I'll fix that up.

Fully agreed. What we have now is OK for a few kernel/qemu setups, but clearly does not scale, and will have to be improved.

Indeed. Patch welcome! :-)

Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

This commit adds a few Buildroot "core" tests, testing functionalities such as:

- post-build and post-image scripts
- root filesystem overlays
- timezone support

---
 support/testing/tests/core/__init__.py             |  0
 support/testing/tests/core/post-build.sh           | 12 ++++
 support/testing/tests/core/post-image.sh           | 12 ++++
 .../testing/tests/core/rootfs-overlay1/test-file1  |  1 +
 .../tests/core/rootfs-overlay2/etc/test-file2      |  1 +
 support/testing/tests/core/test_post_scripts.py    | 40 +++++++++++++
 support/testing/tests/core/test_rootfs_overlay.py  | 27 +++++++++
 support/testing/tests/core/test_timezone.py        | 66 ++++++++++++++++++++++
 8 files changed, 159 insertions(+)
 create mode 100644 support/testing/tests/core/__init__.py
 create mode 100755 support/testing/tests/core/post-build.sh
 create mode 100755 support/testing/tests/core/post-image.sh
 create mode 100644 support/testing/tests/core/rootfs-overlay1/test-file1
 create mode 100644 support/testing/tests/core/rootfs-overlay2/etc/test-file2
 create mode 100644 support/testing/tests/core/test_post_scripts.py
 create mode 100644 support/testing/tests/core/test_rootfs_overlay.py
 create mode 100644 support/testing/tests/core/test_timezone.py

new file mode 100644
index 0000000..e69de29

new file mode 100755
index 0000000..b1fb834
@@ -0,0 +1,12 @@
+#!/bin/sh
+(
+printf "arg1,%s\n" "${1}"
+printf "arg2,%s\n" "${2}"
+printf "arg3,%s\n" "${3}"
+printf "TARGET_DIR,%s\n" "${TARGET_DIR}"
+printf "BUILD_DIR,%s\n" "${BUILD_DIR}"
+printf "HOST_DIR,%s\n" "${HOST_DIR}"
+printf "STAGING_DIR,%s\n" "${STAGING_DIR}"
+printf "BINARIES_DIR,%s\n" "${BINARIES_DIR}"
+printf "BR2_CONFIG,%s\n" "${BR2_CONFIG}"
+) > ${BUILD_DIR}/post-build.log

new file mode 100755
index 0000000..1c2a0f2
@@ -0,0 +1,12 @@
+#!/bin/sh
+(
+printf "arg1,%s\n" "${1}"
+printf "arg2,%s\n" "${2}"
+printf "arg3,%s\n" "${3}"
+printf "TARGET_DIR,%s\n" "${TARGET_DIR}"
+printf "BUILD_DIR,%s\n" "${BUILD_DIR}"
+printf "HOST_DIR,%s\n" "${HOST_DIR}"
+printf "STAGING_DIR,%s\n" "${STAGING_DIR}"
+printf "BINARIES_DIR,%s\n" "${BINARIES_DIR}"
+printf "BR2_CONFIG,%s\n" "${BR2_CONFIG}"
+) > ${BUILD_DIR}/post-image.log

new file mode 100644
index 0000000..323fae0
@@ -0,0 +1 @@
+foobar

new file mode 100644
index 0000000..34a3ec2
@@ -0,0 +1 @@
+barfoo

new file mode 100644
index 0000000..d2a0be1
@@ -0,0 +1,40 @@
+import os
+import csv
+
+import infra.basetest
+
+class TestPostScripts(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_INIT_NONE=y
+BR2_SYSTEM_BIN_SH_NONE=y
+# BR2_PACKAGE_BUSYBOX is not set
+BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+BR2_ROOTFS_POST_IMAGE_SCRIPT="{}"
+BR2_ROOTFS_POST_SCRIPT_ARGS="foobar baz"
+""".format(infra.filepath("tests/core/post-build.sh"),
+           infra.filepath("tests/core/post-image.sh"))
+
+    def check_post_log_file(self, path, what):
+        lines = {}
+        with open(path, 'rb') as csvfile:
+            r = csv.reader(csvfile, delimiter=',')
+            for row in r:
+                lines[row[0]] = row[1]
+
+        self.assertEqual(lines["arg1"], os.path.join(self.builddir, what))
+        self.assertEqual(lines["arg2"], "foobar")
+        self.assertEqual(lines["arg3"], "baz")
+        self.assertEqual(lines["TARGET_DIR"], os.path.join(self.builddir, "target"))
+        self.assertEqual(lines["BUILD_DIR"], os.path.join(self.builddir, "build"))
+        self.assertEqual(lines["HOST_DIR"], os.path.join(self.builddir, "host"))
+        staging = os.readlink(os.path.join(self.builddir, "staging"))
+        self.assertEqual(lines["STAGING_DIR"], staging)
+        self.assertEqual(lines["BINARIES_DIR"], os.path.join(self.builddir, "images"))
+        self.assertEqual(lines["BR2_CONFIG"], os.path.join(self.builddir, ".config"))
+
+    def test_run(self):
+        f = os.path.join(self.builddir, "build", "post-build.log")
+        self.check_post_log_file(f, "target")
+        f = os.path.join(self.builddir, "build", "post-image.log")
+        self.check_post_log_file(f, "images")

new file mode 100644
index 0000000..42d890b
@@ -0,0 +1,27 @@
+import os
+import subprocess
+
+import infra.basetest
+
+def compare_file(file1, file2):
+    return subprocess.call(["cmp", file1, file2])
+
+class TestRootfsOverlay(infra.basetest.BRTest):
+
+    rootfs_overlay_path = infra.filepath("tests/core/rootfs-overlay")
+    rootfs_overlay = "BR2_ROOTFS_OVERLAY=\"{0}1 {0}2\"".format(rootfs_overlay_path)
+
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+            infra.basetest.MINIMAL_CONFIG + \
+            rootfs_overlay
+
+    def test_run(self):
+        target_file = os.path.join(self.builddir, "target", "test-file1")
+        overlay_file = "{}1/test-file1".format(self.rootfs_overlay_path)
+        ret = compare_file(overlay_file, target_file)
+        self.assertEqual(ret, 0)
+
+        target_file = os.path.join(self.builddir, "target", "etc", "test-file2")
+        overlay_file = "{}2/etc/test-file2".format(self.rootfs_overlay_path)
+        ret = compare_file(overlay_file, target_file)
+        self.assertEqual(ret, 0)

2.7.4

Two minor comments:

I think it would be cleaner to move support files for tests, like the post-build.sh and post-image.sh and rootfs-overlays, to a subdirectory instead of in the same location as the tests.

Also, I would argue that the rootfs-overlays do not need to be real files in git but can be generated on the fly during the test.

/Thomas

Hello,

Can you suggest a name for this subdirectory ? :-)

I tried to do that, but:

1. It complicates quite a bit the test scenario.

2. There are plenty of other test cases where we will really need some data that cannot be generated: a set of patches to test if patches on packages are properly applied, a set of dummy BR2_EXTERNAL trees, etc. So we will anyway have some "supporting" artifacts in the tree, and therefore I thought it wasn't worth the effort making this test more complicated.

Yann did the exact same comment while reviewing the previous version, so I tried generating the files automatically, and went through this reasoning before giving up and keeping things as they are.

Best regards,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

support_files
test_data
?

Ok, I see. No issue...

/Thomas

This commit adds a number of test cases for various filesystem formats:
ext2/3/4, iso9660, jffs2, squashfs, ubi/ubifs and yaffs2. All of them except yaffs2 are runtime tested. The iso9660 set of test cases is particularly rich, testing the proper operation of the iso9660 support with all of grub, grub2 and isolinux.

---
 support/testing/conf/grub-menu.lst                 |  20 +++
 support/testing/conf/grub2.cfg                     |   7 +
 support/testing/conf/isolinux.cfg                  |   5 +
 .../testing/conf/minimal-x86-qemu-kernel.config    |  23 +++
 support/testing/tests/fs/__init__.py               |   0
 support/testing/tests/fs/test_ext.py               | 119 +++++++++++++++
 support/testing/tests/fs/test_iso9660.py           | 162 +++++++++++++++++++++
 support/testing/tests/fs/test_jffs2.py             |  45 ++++++
 support/testing/tests/fs/test_squashfs.py          |  37 +++++
 support/testing/tests/fs/test_ubi.py               |  39 +++++
 support/testing/tests/fs/test_yaffs2.py            |  12 ++
 11 files changed, 469 insertions(+)
 create mode 100644 support/testing/conf/grub-menu.lst
 create mode 100644 support/testing/conf/grub2.cfg
 create mode 100644 support/testing/conf/isolinux.cfg
 create mode 100644 support/testing/conf/minimal-x86-qemu-kernel.config
 create mode 100644 support/testing/tests/fs/__init__.py
 create mode 100644 support/testing/tests/fs/test_ext.py
 create mode 100644 support/testing/tests/fs/test_iso9660.py
 create mode 100644 support/testing/tests/fs/test_jffs2.py
 create mode 100644 support/testing/tests/fs/test_squashfs.py
 create mode 100644 support/testing/tests/fs/test_ubi.py
 create mode 100644 support/testing/tests/fs/test_yaffs2.py

new file mode 100644
index 0000000..6143d80
@@ -0,0 +1,20 @@
+default		0
+timeout		1
+
+# Used when no splashimage is used
+color 		cyan/blue white/blue
+
+# Gets enabled/disabled depending on Grub support for splashimage
+splashimage	/boot/grub/splash.xpm.gz
+
+# Used when a splashimage is enabled
+foreground 	000000
+background 	cccccc
+
+title		Buildroot ISO9660 image
+kernel		__KERNEL_PATH__ root=/dev/sr0 console=ttyS0,115200
+initrd		__INITRD_PATH__
+
+title		Hard Drive (first partition)
+rootnoverify	(hd0)
+chainloader	+1

new file mode 100644
index 0000000..a982d0b
@@ -0,0 +1,7 @@
+set default="0"
+set timeout="1"
+
+menuentry "Buildroot" {
+	linux __KERNEL_PATH__ root=/dev/sr0 console=ttyS0,115200
+	initrd __INITRD_PATH__
+}

new file mode 100644
index 0000000..ba031a6
@@ -0,0 +1,5 @@
+default 1
+label 1
+      kernel __KERNEL_PATH__
+      initrd __INITRD_PATH__
+      append root=/dev/sr0 console=ttyS0,115200

new file mode 100644
index 0000000..8f6ceef
@@ -0,0 +1,23 @@
+# CONFIG_64BIT is not set
+CONFIG_SYSVIPC=y
+CONFIG_BLK_DEV_INITRD=y
+# CONFIG_SUSPEND is not set
+# CONFIG_ACPI is not set
+CONFIG_CPU_IDLE=y
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_BLK_DEV_SD=y
+CONFIG_BLK_DEV_SR=y
+CONFIG_ATA=y
+CONFIG_ATA_PIIX=y
+CONFIG_INPUT_EVDEV=y
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_POWER_SUPPLY=y
+CONFIG_THERMAL=y
+CONFIG_EXT4_FS=y
+CONFIG_ISO9660_FS=y
+CONFIG_JOLIET=y
+CONFIG_TMPFS=y
+CONFIG_TMPFS_POSIX_ACL=y
+# CONFIG_VIRTUALIZATION is not set

new file mode 100644
index 0000000..e69de29

new file mode 100644
index 0000000..edaa087
@@ -0,0 +1,37 @@
+import os
+import subprocess
+
+import infra.basetest
+
+class TestSquashfs(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_SQUASHFS=y
+# BR2_TARGET_ROOTFS_SQUASHFS4_GZIP is not set
+BR2_TARGET_ROOTFS_SQUASHFS4_LZ4=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+    def test_run(self):
+        unsquashfs_cmd = ["host/usr/bin/unsquashfs", "-s", "images/rootfs.squashfs"]
+        out = subprocess.check_output(unsquashfs_cmd,
+                                      cwd=self.builddir,
+                                      env={"LANG": "C"})
+        out = out.splitlines()
+        self.assertEqual(out[0],
+                         "Found a valid SQUASHFS 4:0 superblock on images/rootfs.squashfs.")
+        self.assertEqual(out[3], "Compression lz4")
+
+        img = os.path.join(self.builddir, "images", "rootfs.squashfs")
+        subprocess.call(["truncate", "-s", "%1M", img])
+
+        self.emulator.boot(arch="armv7",
+                           kernel="builtin",
+                           kernel_cmdline=["root=/dev/mmcblk0",
+                                           "rootfstype=squashfs"],
+                           options=["-drive", "file={},if=sd,format=raw".format(img)])
+        self.emulator.login()
+
+        cmd = "mount | grep '/dev/root on / type squashfs'"
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)

new file mode 100644
index 0000000..ede4999
@@ -0,0 +1,39 @@
+import subprocess
+import os
+
+import infra.basetest
+
+class TestUbi(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_UBIFS=y
+BR2_TARGET_ROOTFS_UBIFS_LEBSIZE=0x7ff80
+BR2_TARGET_ROOTFS_UBIFS_MINIOSIZE=0x1
+BR2_TARGET_ROOTFS_UBI=y
+BR2_TARGET_ROOTFS_UBI_PEBSIZE=0x80000
+BR2_TARGET_ROOTFS_UBI_SUBSIZE=1
+"""
+
+    # TODO: if you boot Qemu twice on the same UBI image, it fails to
+    # attach the image the second time, with "ubi0 error:
+    # ubi_read_volume_table: the layout volume was not found".
+    # To be investigated.
+    def test_run(self):
+        img = os.path.join(self.builddir, "images", "rootfs.ubi")
+        out = subprocess.check_output(["file", img],
+                                      cwd=self.builddir,
+                                      env={"LANG": "C"})
+        out = out.splitlines()
+
+        subprocess.call(["truncate", "-s 128M", img])
+
+        self.emulator.boot(arch="armv7",
+                           kernel="builtin",
+                           kernel_cmdline=["root=ubi0:rootfs",
+                                           "ubi.mtd=0",
+                                           "rootfstype=ubifs"],
+                           options=["-drive", "file={},if=pflash".format(img)])
+        self.emulator.login()
+        cmd = "mount | grep 'ubi0:rootfs on / type ubifs'"
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)

new file mode 100644
index 0000000..0ffb758
@@ -0,0 +1,12 @@
+import os
+
+import infra.basetest
+
+class TestYaffs2(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+            infra.basetest.MINIMAL_CONFIG + \
+            "BR2_TARGET_ROOTFS_YAFFS2=y"
+
+    def test_run(self):
+        img = os.path.join(self.builddir, "images", "rootfs.yaffs2")
+        self.assertTrue(os.path.exists(img))

2.7.4

Hi,

this the review I had written a few days ago. Since the patch has just been applied, I've reworded it as a list of proposed improvements and request for clarifications.

I suspect it's already been suggested, but I would rather put these files in the 'fs' subdirectory, unless we expect to use them for non-fs tests.

Do other people agree? Should I send a patch?

[...]

...and here it is reassigned to a list (the line fields, as they are called in awk. Or is the second assignment of 'lines' creating a new variable with the same name? Whether the reply, I don't like this.

    for line in out:
        fields = line.split(": ")
        if fields[0] == prop:
            return fields[1].strip()

[...]

[...]

Luca

This commit adds some basic tests for two Buildroot packages: python and dropbear. These tests are by no mean meant to be exhaustive, but mainly to serve as initial examples for other tests.

---
 support/testing/tests/package/__init__.py      |  0
 support/testing/tests/package/test_dropbear.py | 28 +++++++++++++++++++++
 support/testing/tests/package/test_python.py   | 35 ++++++++++++++++++++++++++
 3 files changed, 63 insertions(+)
 create mode 100644 support/testing/tests/package/__init__.py
 create mode 100644 support/testing/tests/package/test_dropbear.py
 create mode 100644 support/testing/tests/package/test_python.py

new file mode 100644
index 0000000..e69de29

new file mode 100644
index 0000000..a64b782
@@ -0,0 +1,28 @@
+import os
+
+import infra.basetest
+
+class TestDropbear(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_GENERIC_ROOT_PASSWD="testpwd"
+BR2_SYSTEM_DHCP="eth0"
+BR2_PACKAGE_DROPBEAR=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+    def test_run(self):
+        img = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-initrd", img,
+                                    "-net", "nic",
+                                    "-net", "user,hostfwd=tcp::2222-:22"])
+        self.emulator.login("testpwd")
+        cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:22"
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+        # Would be useful to try to login through SSH here, through
+        # localhost:2222, though it is not easy to pass the ssh
+        # password on the command line.

new file mode 100644
index 0000000..5532fb5
@@ -0,0 +1,35 @@
+import os
+
+import infra.basetest
+
+class TestPythonBase(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_PACKAGE_PYTHON=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+    def test_run(self):
+        cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-initrd", cpio_file])
+        self.emulator.login()
+        cmd = "python --version 2>&1 | grep '^Python 2'"
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+        cmd = "python -c 'import math; math.floor(12.3)'"
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+        cmd = "python -c 'import ctypes;"
+        cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
+        cmd += "print libc.time(None)'"
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+        cmd = "python -c 'import zlib'"
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 1)

2.7.4

Hello,

Good idea. Let's leave this as a future improvement :-)

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

This commit adds an initial toolchain test case, testing the ARM CodeSourcery toolchain, just checking that the proper sysroot is used, and that a minimal Linux system boots fine under Qemu.

---
 support/testing/tests/toolchain/__init__.py      |   0
 support/testing/tests/toolchain/test_external.py | 156 +++++++++++++++++++++++
 2 files changed, 156 insertions(+)
 create mode 100644 support/testing/tests/toolchain/__init__.py
 create mode 100644 support/testing/tests/toolchain/test_external.py

new file mode 100644
index 0000000..e69de29

2.7.4

Hi,

Luca

I gave some minor comments/suggestions, but nothing blocking. I fully agree to not hold on to this much longer...

Well done!

/Thomas

Hello,

Since this has been around for a long time, I've finally applied this series.

I know Luca, Ricardo and Thomas DS had some comments, but I believe it would be easier to address them as follow-up patches. So Luca, Ricardo, Thomas, your patches to improve this testing infrastructure are more than welcome. I'm sure it needs to be extended and improved in many different ways to test more Buildroot functionalities.

Best regards,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Hi,

Bad timing on my side: I was reviewing all this work in detail a couple of days ago, but my reviews to 2 patches are in my drafts folder waiting for the remaining ones... Ok, I'll send them anyway right noe as a list of proposed improvements.

I totally agree this is the best way to proceed. Now everybody can easily send small, incremental patches to improve the work. Thanks for having applied these patches.

Any plan to have the tests run automatically (on the autobuilders maybe)?

Luca

Hello,

Thanks for your review!

Yes, it will allow everyone to send small, incremental patches, me included.

Yes, we will definitely want to do that. I don't think the autobuilders will do it. Instead I believe running the tests on Gitlab CI would be easier and more appropriate.

I'm sure Arnout can help :-)

Best regards,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com