Adds test infrastructure for packages

This patch series adds a test generation infrastructure for packages to Buildroot.
The generated tests are robotframework tests.

While it is always necessary to test that packages are installed correctly and work in an intended way, Buildroot does not currently provide any tests of package nor any way to automate the process of creating tests depending on your configuration or even share tests you have written for packages. You have to write tests manually almost everytime you start a new project with buildroot. This can cost a lot of time.

This patch series is intented as a solution for that problem.

Why Robotframework Tests:
Robotframework is a widely used test automation framework that allow writing tests in various languages (Python, Java and even C), provides useful test librairies and allow writing tests in various syntaxes. Robotframework tests are meant to be easy to read by people whom did not write the tests, and that can definitly be useful.

List of patches from the series :
* Patch 1/6 adds a test generation step before install target step.
* Patch 2/6 adds useful keywords for testing through Telnet and Qemu
* Patch 3/6 adds menuconfig options to ease the configuration required for testing through telnet and Qemu
* Patch 4/6 adds documentation for patches 1,2 and 3
* Patch 5/6 adds exemple tests for package python-flask
* patch 6/6 adds exemple tests for package python (inspired by tests from https://github.com/tpetazzoni/buildroot-runtime-test)

Denis THULIN (6):
python-robotframework: New package
Adds package test infrastructure
tests: create variable files through kconfig
tests: Adds user manual entry
flask: Adds robotframework tests
python: Adds tests

 Config.in                                          |   6 +
 Makefile                                           |   9 +-
 docs/manual/common-usage.txt                       |   2 +
 docs/manual/test-infrastructure.txt                | 239 +++++++++++++++++++++
 package/Config.in                                  |   1 +
 package/Config.in.host                             |   1 +
 package/Makefile.in                                |   1 +
 package/pkg-generic.mk                             |  20 +-
 package/pkg-test.mk                                |  94 ++++++++
 package/python-flask/python-flask.mk               |   5 +
 .../target_test_material/target_test.robot         |  25 +++
 package/python-robotframework/Config.in            |   9 +
 package/python-robotframework/Config.in.host       |   8 +
 .../python-robotframework.hash                     |   4 +
 .../python-robotframework/python-robotframework.mk |  16 ++
 package/python/python.mk                           |  17 ++
 package/python/python.robot                        |  32 +++
 tests/Config.in                                    |   8 +
 tests/host/resource.robot                          |   7 +
 tests/target/resource.robot                        |   1 +
 tests/tests.mk                                     |  20 ++
 21 files changed, 522 insertions(+), 3 deletions(-)
 create mode 100644 docs/manual/test-infrastructure.txt
 create mode 100644 package/pkg-test.mk
 create mode 100644 package/python-flask/target_test_material/target_test.robot
 create mode 100644 package/python-robotframework/Config.in
 create mode 100644 package/python-robotframework/Config.in.host
 create mode 100644 package/python-robotframework/python-robotframework.hash
 create mode 100644 package/python-robotframework/python-robotframework.mk
 create mode 100644 package/python/python.robot
 create mode 100644 tests/Config.in
 create mode 100644 tests/host/resource.robot
 create mode 100644 tests/target/resource.robot
 create mode 100644 tests/tests.mk

2.5.0

A generic test automation framework written in python.

---
v0:
- Generated using python-package-generator.py
- Dependencies on python-zlib and python-pyexpat were added manually
- Modification of packages/Config.in done manually.
v1:
- fixed indentation error on dependencies (was 4 spaces instead of tab)
- Changed hash from md5 to sha256 (modification of scanpypi)
- Fixed typos (modification of scanpypi)
v2:
- Added the md5 hash from pypi
- Fixed typos
v3:
- Changed the hash file

---
 package/Config.in                                        |  1 +
 package/Config.in.host                                   |  1 +
 package/python-robotframework/Config.in                  |  9 +++++++++
 package/python-robotframework/Config.in.host             |  8 ++++++++
 package/python-robotframework/python-robotframework.hash |  4 ++++
 package/python-robotframework/python-robotframework.mk   | 16 ++++++++++++++++
 6 files changed, 39 insertions(+)
 create mode 100644 package/python-robotframework/Config.in
 create mode 100644 package/python-robotframework/Config.in.host
 create mode 100644 package/python-robotframework/python-robotframework.hash
 create mode 100644 package/python-robotframework/python-robotframework.mk

index 783fbb4..c7fadd6 100644
@@ -656,6 +656,7 @@ menu "external python modules"
 	source "package/python-pyyaml/Config.in"
 	source "package/python-pyzmq/Config.in"
 	source "package/python-requests/Config.in"
+	source "package/python-robotframework/Config.in"
 	source "package/python-rtslib-fb/Config.in"
 	source "package/python-serial/Config.in"
 	source "package/python-setuptools/Config.in"

index 1e047aa..4526892 100644
@@ -22,6 +22,7 @@ menu "Host utilities"
 	source "package/patchelf/Config.in.host"
 	source "package/pwgen/Config.in.host"
 	source "package/qemu/Config.in.host"
+	source "package/python-robotframework/Config.in.host"
 	source "package/sam-ba/Config.in.host"
 	source "package/squashfs/Config.in.host"
 	source "package/sunxi-tools/Config.in.host"

new file mode 100644
index 0000000..9523893
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+	bool "python-robotframework"
+	depends on BR2_PACKAGE_PYTHON
+	select BR2_PACKAGE_PYTHON_ZLIB
+	select BR2_PACKAGE_PYTHON_PYEXPAT
+	help
+	  A generic test automation framework
+
+	  http://robotframework.org

new file mode 100644
index 0000000..c1f5883
@@ -0,0 +1,8 @@
+config BR2_PACKAGE_HOST_PYTHON_ROBOTFRAMEWORK
+	bool "host python-robotframework"
+	depends on BR2_PACKAGE_HOST_PYTHON
+	help
+	  A generic test automation framework used for testing buildroot
+	  packages
+
+	  http://robotframework.org

new file mode 100644
index 0000000..58ff12d
@@ -0,0 +1,4 @@
+# md5 from https://pypi.python.org/pypi/robotframework/json
+md5	db4a3330dfaa761c1599ca1064396eee  robotframework-2.9.tar.gz
+# sha256 calculated by scanpypi
+sha256	13a0340864e8c7735302ce39f8d3f9d01bed7db7c055e55043396242a615d8f1  robotframework-2.9.tar.gz

new file mode 100644
index 0000000..afef695
@@ -0,0 +1,16 @@
+################################################################################
+#
+# python-robotframework
+#
+################################################################################
+
+PYTHON_ROBOTFRAMEWORK_VERSION = 2.9
+PYTHON_ROBOTFRAMEWORK_SOURCE = robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
+PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework
+PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = setuptools
+PYTHON_ROBOTFRAMEWORK_LICENSE = Apache-2.0
+PYTHON_ROBOTFRAMEWORK_LICENSE_FILES = LICENSE.txt
+
+
+$(eval $(python-package))
+$(eval $(host-python-package))

2.5.0

This patch adds a test generation step in pkg-generic.mk.

The tests are generated used various information found in the package's .mk file. Tests can be written either directly inside the .mk file or in a separate file.
The test are written for robot-framework.

The create-test step is an order only dependency of TARGET_INSTALL_TARGET. It requires configure to be done. There are two kinds of tests:
- host tests: Tests that are ran directly on the output/target directory. Those test are mostly meant to make sure that some files
- target tests: Tests that are meant to be ran from a running buildroot on itself. Those need to be copied manually inside the the rootfs and require package python-robotframework (And python 2 as robotframework does not work with python 3).

For any kind of test, you can put global test resources (Keywords, Variables, Libraries) inside the support/test/<test-kind>/ folder. Those can be used from any test of this kind.

---

---
 Makefile                    |  9 ++++-
 package/Makefile.in         |  1 +
 package/pkg-generic.mk      | 20 +++++++++-
 package/pkg-test.mk         | 94 +++++++++++++++++++++++++++++++++++++++++++++
 tests/host/resource.robot   |  7 ++++
 tests/target/resource.robot |  1 +
 tests/tests.mk              | 17 ++++++++
 7 files changed, 146 insertions(+), 3 deletions(-)
 create mode 100644 package/pkg-test.mk
 create mode 100644 tests/host/resource.robot
 create mode 100644 tests/target/resource.robot
 create mode 100644 tests/tests.mk

index 07fd69b..63c27c1 100644
@@ -196,6 +196,7 @@ TARGET_DIR := $(BASE_DIR)/target
 # .config. HOST_DIR will be overwritten later when .config is included.
 HOST_DIR := $(BASE_DIR)/host
 GRAPHS_DIR := $(BASE_DIR)/graphs
+TEST_DIR := $(BASE_DIR)/tests
 
 LEGAL_INFO_DIR = $(BASE_DIR)/legal-info
 REDIST_SOURCES_DIR_TARGET = $(LEGAL_INFO_DIR)/sources

@@ -422,6 +423,8 @@ include fs/common.mk
 
 include $(BR2_EXTERNAL)/external.mk
 
+include tests/tests.mk
+
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR)
 

@@ -556,7 +559,7 @@ endif
 
 $(TARGETS_ROOTFS): target-finalize
 
-target-finalize: $(PACKAGES)
+target-finalize: $(PACKAGES) tests-common-resources
 	@$(call MESSAGE,"Finalizing target directory")
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \

@@ -623,6 +626,7 @@ endif
 		$(call MESSAGE,"Executing post-build script $(s)"); \
 		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
+
 target-post-image: $(TARGETS_ROOTFS) target-finalize
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \

@@ -825,7 +829,8 @@ printvars:
 clean:
 	rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(BUILD_DIR) $(BASE_DIR)/staging \
-		$(LEGAL_INFO_DIR) $(GRAPHS_DIR)
+		$(LEGAL_INFO_DIR) $(GRAPHS_DIR) \
+		$(TEST_DIR)
 
 distclean: clean
 ifeq ($(DL_DIR),$(TOPDIR)/dl)

index 545694f..68a970a 100644
@@ -400,3 +400,4 @@ include package/pkg-generic.mk
 include package/pkg-kconfig.mk
 include package/pkg-rebar.mk
 include package/pkg-kernel-module.mk
+include package/pkg-test.mk

index 6a7d97e..f688219 100644
@@ -222,6 +222,17 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 	$(Q)touch $@
 	@$(call step_end,install-staging)
 
+# Create corresponding tests
+$(BUILD_DIR)/%/.stamp_tests_created:
+	$(call step_start,create-tests)
+	$(foreach hook,$($(PKG)_PRE_CREATE_TESTS_HOOKS),$(call $(hook))$(sep))
+	@$(call MESSAGE,"Creating corresponding tests")
+	$(call TEST_BUILD_ALL)
+	$(foreach hook,$($(PKG)_POST_CREATE_TESTS_HOOKS),$(call $(hook))$(sep))
+	$(Q)touch $@
+	@$(call step_end,create-tests)
+
+
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
 	@$(call step_start,install-image)

@@ -472,7 +483,7 @@ $(2)_TARGET_PATCH =		$$($(2)_DIR)/.stamp_patched
 $(2)_TARGET_EXTRACT =		$$($(2)_DIR)/.stamp_extracted
 $(2)_TARGET_SOURCE =		$$($(2)_DIR)/.stamp_downloaded
 $(2)_TARGET_DIRCLEAN =		$$($(2)_DIR)/.stamp_dircleaned
-
+$(2)_CREATE_TESTS =	$$($(2)_DIR)/.stamp_tests_created
 # default extract command
 $(2)_EXTRACT_CMDS ?= \
 	$$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \

@@ -563,6 +574,9 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  configure
 $$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_PATCH)
 
+$(1)-create-tests:		$$($(2)_CREATE_TESTS)
+$$($(2)_TARGET_INSTALL_TARGET):	| $$($(2)_CREATE_TESTS)
+
 $(1)-patch:		$$($(2)_TARGET_PATCH)
 $$($(2)_TARGET_PATCH):	$$($(2)_TARGET_EXTRACT)
 # Order-only dependency

@@ -663,6 +677,9 @@ $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
 $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
+$$($(2)_CREATE_TESTS):			PKG=$(2)
+$$($(2)_CREATE_TESTS):			pkg=$(1)
+$$($(2)_CREATE_TESTS):			ORIGINAL_DIR=$(pkgdir)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
 $$($(2)_TARGET_RSYNC):                  SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
 $$($(2)_TARGET_RSYNC):                  PKG=$(2)

@@ -833,6 +850,7 @@ endif
 	$(1)-graph-depends \
 	$(1)-install \
 	$(1)-install-host \
+	$(1)-create-tests \
 	$(1)-install-images \
 	$(1)-install-staging \
 	$(1)-install-target \

new file mode 100644
index 0000000..650d4c8
@@ -0,0 +1,7 @@
+*** Settings ***
+Library         OperatingSystem
+Variables       common_variables.py
+*** Keywords ***
+Package Installed On Target
+    [arguments]    ${File}
+    OperatingSystem.Should Exist    ${target_dir}/${File}

new file mode 100644
index 0000000..c742ffc
@@ -0,0 +1 @@
+*** Variables ***

new file mode 100644
index 0000000..a246c6e
@@ -0,0 +1,17 @@
+tests-common-resources: host-python-robotframework
+	rm -rf $(TEST_DIR)/host/resources
+	rm -rf $(TEST_DIR)/target/resources
+	mkdir -p $(TEST_DIR)/host
+	cp -r tests/host $(TEST_DIR)/host/resources
+	echo target_dir = \"$(TARGET_DIR)\" >> $(TEST_DIR)/host/resources/common_variables.py
+	mkdir -p $(TEST_DIR)/target
+	cp -r tests/target $(TEST_DIR)/target/resources
+
+
+PYBOT = $(HOST_DIR)/usr/bin/pybot
+PYBOT_OPTS ?= -d $(TEST_DIR)/logs -x xunit.xml
+
+tests: tests-common-resources
+	@$(PYBOT) $(PYBOT_OPTS) $(TEST_DIR)/host
+
+.PHONY: test-common-resources tests

2.5.0

Install of target tests can now be selected from menuconfig This patch adds a new menu entry in Config.in in order to configure the test environnment.

---

---
 Config.in       | 6 ++++++
 tests/Config.in | 8 ++++++++
 tests/tests.mk  | 3 +++
 3 files changed, 17 insertions(+)
 create mode 100644 tests/Config.in

index 8b1bd8d..51e0ea2 100644
@@ -624,3 +624,9 @@ menu "User-provided options"
 source "$BR2_EXTERNAL/Config.in"
 
 endmenu
+
+menu "Test Configuration Settings"
+
+source "tests/Config.in"
+
+endmenu

new file mode 100644
index 0000000..937753c
@@ -0,0 +1,8 @@
+comment "Target tests require robotframework"
+	depends on !BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+
+config BR2_TEST_TARGET
+	bool "Install tests to target"
+	depends on BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+	help
+	  Enable this if you want to install tests on your target

index a246c6e..ad05f7c 100644
@@ -6,6 +6,9 @@ tests-common-resources: host-python-robotframework
 	echo target_dir = \"$(TARGET_DIR)\" >> $(TEST_DIR)/host/resources/common_variables.py
 	mkdir -p $(TEST_DIR)/target
 	cp -r tests/target $(TEST_DIR)/target/resources
+ifeq ($(BR2_TEST_TARGET),y)
+	cp -r $(TEST_DIR)/target $(TARGET_DIR)/usr/share/buildroot_tests
+endif
 
 
 PYBOT = $(HOST_DIR)/usr/bin/pybot

2.5.0

Adds user manual section about how to use and write tests

The documentation covers everything from writing your own tests to setting up your configuration to test throught telnet and qemu.

---

---
 docs/manual/common-usage.txt        |   2 +
 docs/manual/test-infrastructure.txt | 239 ++++++++++++++++++++++++++++++++++++
 2 files changed, 241 insertions(+)
 create mode 100644 docs/manual/test-infrastructure.txt

index 5b27b1f..69c90b6 100644
@@ -276,3 +276,5 @@ BR2_GRAPH_OUT=png make graph-build
 include::eclipse-integration.txt[]
 
 include::advanced.txt[]
+
+include::test-infrastructure.txt[]

2.5.0

Adds a 'host' test to check that the flask folder is created, a 'target' test that checks that flask runs correctly

The 'host' test is ran on the target folder, it merely checks that the flask folder exists

The 'target' test starts a flask process on a specified port and localhost

---

---
 package/python-flask/python-flask.mk               |  5 +++++
 .../target_test_material/target_test.robot         | 25 ++++++++++++++++++++++
 2 files changed, 30 insertions(+)
 create mode 100644 package/python-flask/target_test_material/target_test.robot

index 8aa675e..18dc846 100644
@@ -12,4 +12,9 @@ PYTHON_FLASK_LICENSE = BSD-3c
 PYTHON_FLASK_LICENSE_FILES = LICENSE
 PYTHON_FLASK_DEPENDENCIES = python-werkzeug python-jinja2 python-itsdangerous
 
+
+PYTHON_FLASK_TEST_INSTALL_LOCATION = /usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/flask
+
+PYTHON_FLASK_TEST_MATERIAL = target_test_material
+
 $(eval $(python-package))

new file mode 100644
index 0000000..78f4d38
@@ -0,0 +1,25 @@
+*** Settings ***
+Documentation   This is a simple test suite meant to make sure that flask works
+...             properly. Please ensure that no other program is using tcp port
+...             5000 before running it.
+Library         Process
+*** Variables ***
+${flask_range}=         "127.0.0.1"
+${flask_port}=          5000
+${flask_address}=       "127.0.0.1"
+*** Test Cases ***
+Test Port
+    [setup]         Start Flask With Hello World
+    [Teardown]      Kill Flask
+    Wait Until Keyword Succeeds      2 sec    10 sec     Wget Returns OK
+*** Keywords ***
+Start Flask With Hello World
+    ${handle}=    Start Process     python2     -c     from flask import Flask; app \= Flask(__name__); app.route(\"/\")(lambda: \"hello world\"); app.run(host\=${flask_range},port\=${flask_port})
+    Set Suite Variable    ${HANDLE}    ${handle}
+
+Kill Flask
+    Terminate Process    ${HANDLE}
+
+Wget Returns Ok
+    ${result}=      Run Process     wget    http://127.0.0.1:5000
+    Should Be Equal As Integers    0   ${result.rc}

2.5.0

Adds a 'host' test (usr/bin/python exists) and a few target tests.

Current target tests:
Check Python Version: Should be 2
Check Python Math: import math module and check that it works Check Python Libc: import ctypes and load libc Check Python Zlib: import zlib
Check Python SSL: import ssl

Tests of zlib and ssl make sure that the result is correct according to the buildroot configuration.

---

---
 package/python/python.mk    | 17 +++++++++++++++++
 package/python/python.robot | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 49 insertions(+)
 create mode 100644 package/python/python.robot

index 80ffbde..047567a 100644
@@ -214,6 +214,23 @@ endif
 # Provided to other packages
 PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/:$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/
 
+# Host Tests
+PYTHON_TEST_INSTALL_LOCATION := /usr/bin/python
+
+# Target tests
+PYTHON_TEST_ORIGINAL = python.robot
+ifeq ($(BR2_PACKAGE_PYTHON_ZLIB),y)
+	PYTHON_TEST_VARIABLES += zlib_return_code=0
+else
+	PYTHON_TEST_VARIABLES += zlib_return_code=1
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON_SSL),y)
+	PYTHON_TEST_VARIABLES += ssl_return_code=0
+else
+	PYTHON_TEST_VARIABLES += ssl_return_code=1
+endif
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 

new file mode 100644
index 0000000..9ea3203
@@ -0,0 +1,32 @@
+*** Settings ***
+Documentation       Test suite for package python of buildroot
+Library             Process
+*** Variables ***
+*** Test Cases ***
+Python Version
+    ${result}=              Run Process       python      --version       stderr=STDOUT
+    Should Start With       ${result.stdout}     Python 2
+
+Python Math
+    [Template]      Check Python Return Code of Python Command
+    0       import math; math.floor(12.3)
+
+Python Libc
+    [Template]      Check Python Return Code of Python Command
+    0       import ctypes; libc = ctypes.dll.LoadLibrary(\\"libc.so\\"); print libc.time(None)
+
+Python Zlib
+    [Template]      Check Python Return Code of Python Command
+    ${zlib_return_code}       import zlib
+
+Python SSL
+    [Template]      Check Python Return Code of Python Command
+    ${ssl_return_code}       import ssl
+    
+*** Keywords ***
+Check Python Return Code of Python Command
+    [arguments]     ${return_code}      ${command}
+    ${result}=      Run Process       python      -c      "${command}"
+    Should Be Equal As Integers     ${return_code}      ${result.rc}
+
+

2.5.0

Denis,

Thanks a lot for working on the topic of runtime testing, which is an important topic that needs to be addressed in the Buildroot community.

However, I have to say I personally dislike quite a bit some of the principles of your solutions:

* I don't like the Robot Framework. It requires you to learn another new, specialized (and weird) language to write tests, instead of simply writing them in Python. The Robot Framework is probably good when the people who will write the tests are not developers. But that's not the case of Buildroot users and developers. Many of them probably already know Python (or at least the basics of it), while nobody knows this specialized language. It will be a barrier to getting tests written by Buildroot contributors.

* I don't think tying tests to packages is the correct idea. Some tests need to be done for filesystem formats, some tests for core mechanisms (like checking BR2_EXTERNAL functionality, checking core package infrastructure mechanisms, etc.). I don't think the test mechanism should modify Buildroot itself.

Also, this may be me not understanding properly, but I fail to see where are your test cases. My impression is that you add tests to packages, and then your idea is that whenever a package is enabled in a given configuration, you will run the tests of this package. But who will generate the configurations to be tested? They surely cannot be randomly generated, because many combinations of packages do not make sense (for example a random configuration that has Dropbear and OpenSSH enabled will build fine, but will completely fail at runtime because both will try to bind on port 22).

I certainly do not claim that my proposal at
https://github.com/tpetazzoni/buildroot-runtime-test was ideal and perfect, but it is IMO much more in-line with what we need for
Buildroot: fully written in a language that is generally well-known, and a simple list of test-cases that associate a configuration to be built/executed with a set of actions to be verified.

Best regards,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Hi Thomas,

Thanks for answering that fast.

It seems I the use case for this patch series is not clear. I'm trying to create tests for Buildroot users, those tests are not intended for autobuilds (They could be used, but that was not what I intented when I wrote this patch series).

----- Le 31 Ao? 15, ? 14:48, Thomas Petazzoni thomas.petazzoni at free-electrons.com a ?crit :

There seems to be a big misunderstanding here. This patch series is not meant for the same use case as
https://github.com/tpetazzoni/buildroot-runtime-test or what I wrote at https://github.com/etkaDT/Rfw-buildroot-tests

The test infrastructure is meant to be used by Buildroot users rather than Buildroot devs. The goal here is for users to test if their packages work properply with their Buildroot configuration, not to test if the packages work in general.

In that context, I feel that it is important that anybody can understand the tests results.

Well those tests are not included in the use case I'm working on and I don't think they should be.

You understand how these tests works.
Again, those tests are not meant to work with random configurations but rather with a user's configuration.
In that case, the user will notice that his configuration is not working and will change it so that his packages work correctly.

I agree on that point,
This patch series is just meant for a different use case.

Best regards,
Denis

Hi Denis,

Just to be clear: the intention of this series is to make it possible to
include in the buildroot package a way to specify some tests to run on the host and on the target, where the purpose of these tests is to validate that the package has been built correctly. So it is not exactly testing the buildroot infrastructure (like what Thomas is trying to do), and it's not really running the tests included with the package itself (these are not built and installed), but something in-between.

However, as far as I can see, this patch series only solves a small part of the
problem. It basically creates and installs a bunch of .robot files, and that's it, right? You still have to make the robot framework run on these files. And you still have to express in the robot framework or in jenkins the dependencies to make sure the tests are first built and then executed. Correct?

In addition, it seems to me that the munging done by the TEST_BUILD macro has
little to offer above simply copying the .robot file from the package directory to the test directory. But possibly I don't sufficiently understand what's going on.

And finally, buildroot tries not to enforce any particular policy or tool on
the user. So, much as I like robot, requiring the tests to use this specific framework is definitely a no-go (at least, if it's meant to be used by end users). Especially since it means that a rootfs with python3 can't be tested...

I'm not entirely sure what the way to go is. It definitely is nice to have some
kind of test description as part of the packages - I like that more than the way Thomas did it in buildroot-runtime-test, where the buildroot tree is basically replicated externally for the tests. But the exact location of the test files is just a minor detail I guess.

Bottom line: I'm not in favour of this series either. Of course, the first
patch adding python-robotframework is still relevant independently.

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Hi Arnout,

----- Le 7 Sep 15, ? 0:46, Arnout Vandecappelle arnout at mind.be a ?crit :

Yes.

Yes you do.

It mostly copies the .robot files.
Some tests or robot variables can be written in the .mk file so that they depend on the Buildroot configuration.

Ok.

Yes it is.

Ok, thanks for the answer :)

Regards,
Denis

[snip]

Hi Denis,

We discussed this again at the BR Developer meeting in Dublin, and concluded
that we're going to proceed with the simpler and less-invasive unit-test-based approach. It is in fact already running on Jenkins at [1]. Therefore, I've marked most of the series as Rejected in patchwork.

Regarding python-robotframework, as mentioned before, it is still relevant.
However, neither of the two versions you submitted are suitable at the moment:
- the version of [2] is still using 2.9rc1;
- the version of [3] add a host- version which we don't need. Note that the host-version was not even needed in this series: for something like this, we expect it to be installed on the system rather than building it. So these patches are both marked as Changes Requested.

Regards,
Arnout

[1] http://jenkins.free-electrons.com/job/buildroot-runtime-tests/
[2] http://patchwork.ozlabs.org/patch/501203/
[3] http://patchwork.ozlabs.org/patch/512420/

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF