RFC Adds test infrastructure for packages
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest0 of 6 patches 0 repliesNew
Patches in v1 No applies recorded yet
None of the 6 patches the cover letter announces are in this archive.
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages
Denis,
Quoted 12 lines · Denis THULIN
This patch series adds a test generation infrastructure for packages to Buildroot.
The generated tests are robotframework tests.While it is always necessary to test that packages are installed correctly and work in an intended way, Buildroot does not currently provide any tests of package nor any way to automate the process of creating tests depending on your configuration or even share tests you have written for packages. You have to write tests manually almost everytime you start a new project with buildroot. This can cost a lot of time.
Quoted 1 line
This patch series is intented as a solution for that problem.
Thanks a lot for working on the topic of runtime testing, which is an important topic that needs to be addressed in the Buildroot community.
However, I have to say I personally dislike quite a bit some of the principles of your solutions:
* I don't like the Robot Framework. It requires you to learn another new, specialized (and weird) language to write tests, instead of simply writing them in Python. The Robot Framework is probably good when the people who will write the tests are not developers. But that's not the case of Buildroot users and developers. Many of them probably already know Python (or at least the basics of it), while nobody knows this specialized language. It will be a barrier to getting tests written by Buildroot contributors.
* I don't think tying tests to packages is the correct idea. Some tests need to be done for filesystem formats, some tests for core mechanisms (like checking BR2_EXTERNAL functionality, checking core package infrastructure mechanisms, etc.). I don't think the test mechanism should modify Buildroot itself.
Also, this may be me not understanding properly, but I fail to see where are your test cases. My impression is that you add tests to packages, and then your idea is that whenever a package is enabled in a given configuration, you will run the tests of this package. But who will generate the configurations to be tested? They surely cannot be randomly generated, because many combinations of packages do not make sense (for example a random configuration that has Dropbear and OpenSSH enabled will build fine, but will completely fail at runtime because both will try to bind on port 22).
I certainly do not claim that my proposal at
https://github.com/tpetazzoni/buildroot-runtime-test was ideal and perfect, but it is IMO much more in-line with what we need for
Buildroot: fully written in a language that is generally well-known, and a simple list of test-cases that associate a configuration to be built/executed with a set of actions to be verified.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com