Skip to contentSkip to conversation

Some ideas on open source,> testing

Fuego Agustin Benito Bethencourt 1 participant 1 message Nov 1, 2016 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi

it is my first mail here. Apologies for answering based on a digest mail. I already disabled the digest mode for the future.

Quoted 41 lines

Message: 2
Date: Mon, 24 Oct 2016 17:41:16 +0100
From: Mark Brown <[email protected]>
To: Theodore Ts'o <[email protected]>
Cc: "[email protected]"
<[email protected]>, "[email protected]" <[email protected]>
Subject: Re: [Fuego] [Ksummit-discuss] Some ideas on open source
testing
Message-ID: <[email protected]>
Content-Type: text/plain; charset="us-ascii"

Quoted 11 lines · Theodore Ts'o

And so you might have alarge number of workflows:

* Developers who want a fast smoke test after applying a patch or patch set.
* Developers who want to dig into a specific test failure, and who want to be able to very quickly iterate over running a specific test against a quick succession of test kernels.
* Maintainers who want to run a much more comprehensive test suite that might take hours to run. * Release engineers who want some kind of continuous integration test.

Quoted 2 lines

A test runner framework which is good for one is very likely not going to be good for another.

That's true but there's also an awful lot that can be shared - for example, the mechanics of booting on a given system are going to be the same regardless of how the test was scheduled or how long it will run for. We should be looking for ways to share things as much as we can, especially around the bits where the skills needed to implement align less well with kernel skills.

[UI for reporting]

Quoted 4 lines

I'm hoping to have an intern try to create something like that for gce-xfstests that would run on Google App Engine, over the next couple of months. So maybe by next year I'll have something that we'll be able to show off. We'll see....

Quoted 4 lines

This sort of UI thing seems like a prime example of an area where we could really use some sharing - for example we've got some capacity to run tests in kernelci but nobody to work on UI for doing anything useful with the results.

For UI testing there is an option for systems out there today, openQA, used in production by SUSE and more recently by Red Hat (Fedora).

I would look into it before thinking about extending kernelci to UI testing, so the complexity of what today is successful, among other reasons, for its simplicity.

When it comes to boot testing (acceptance testing) I wonder what is the percentage of cases in which testing on real hardware is required vs testing on emulated environments.

My experience on this has showed me that the percentage I assumed in initially (in theory) was lower than in practice (in favour of emulation). Depending on what is that percentage for you, openQA might be a good enough answer vs extending kernelci/Fuego.

openQA test visualization: https://openqa.opensuse.org/

Link to the project: http://open.qa/

I also believe Fuego can use openQA as a reference for some decisions that need to be taken in the future. There are not that many projects out there for system testing that are fully open, developed and used by people willing to help others.

Best Regards

Signature · 3 lines
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
[email protected]