Skip to contentSkip to conversation

RFC: automated runtime testing on real hardware

DISCUSSION Paul Eggleton 2 participants 4 messages Aug 13, 2013 – Aug 27, 2013

lore
Share & tools
lore
#0 lore raw

Hi all,

As part of the feature development work for 1.5, we've recently added the ability to easily define runtime tests written in python, replacing the old shell-based imagetest.bbclass. However, for 1.5 this is limited to running the tests within a QEMU environment; the obvious and desirable next step is to be able to have these run on real hardware. Clearly this presents some challenges, such as how to manage access to the hardware if you have multiple builders potentially wanting to use the same board at around the same time.

Automated testing on real hardware is something that a lot of us do already, but the likelihood is that we've all been coming up with our own solutions for this, so it would be really great if we could have a single working solution to cover the generally applicable use cases. As a start though it would be interesting to hear from people who have existing test setups or who are generally interested in this area. I'd like to hear feedback on the following:

* What hardware setup do you use for automated testing?

* Do you use any software to manage the hardware? Does this include any
existing open source tools?

* What would you like to see in a common framework to enable this kind of
testing?

For reference, some discussion on this started in another thread [1], but I'd like to throw this out to a more general audience.

Thanks,
Paul

[1] https://lists.yoctoproject.org/pipermail/yocto/2013-August/017687.html

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#1 lore raw
Quoted 10 lines · Paul Eggleton

Hi all,

As part of the feature development work for 1.5, we've recently added the ability to easily define runtime tests written in python, replacing the old shell-based imagetest.bbclass. However, for 1.5 this is limited to running the tests within a QEMU environment; the obvious and desirable next step is to be able to have these run on real hardware. Clearly this presents some challenges, such as how to manage access to the hardware if you have multiple builders potentially wanting to use the same board at around the same time.

Quoted 8 lines

Automated testing on real hardware is something that a lot of us do already, but the likelihood is that we've all been coming up with our own solutions for this, so it would be really great if we could have a single working solution to cover the generally applicable use cases. As a start though it would be interesting to hear from people who have existing test setups or who are generally interested in this area. I'd like to hear feedback on the following:

* What hardware setup do you use for automated testing?

I am starting to put something together using a customized 19" rack along the lines of those used by OSADL for their real-time farm:

https://www.osadl.org/Test-Rack.test-rack.0.html

This is a poor man's BladeCenter for embedded boards in that it provides a common interface to each board for the rack infrastructure. The board specific stuff sits on the trays themselves. The trays are easily removable to take a board to a desk for hands-on work.

A smart-switch, web-power strip, and serial concentrator feed into a "gateway server" where services like conmux glue it all together.

Quoted 3 lines

* Do you use any software to manage the hardware? Does this include any
existing open source tools?

conmux
dnsmasq
udev (for static usb uart naming)

The testrack has some early-stage tools to support it, but I'm still sorting those out and they aren't particularly robust for initial setup quite yet.

Quoted 3 lines

* What would you like to see in a common framework to enable this kind of
testing?

To simplify the board management you allude to above, I believe a pull model, rather than a push, is a better architecture.

Your builders can complete builds and inject test records into a database with a priority. The boards sit in a boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and push the results back to the database. They select tests based on priority, and the builder doesn't have to do anything at all to manage the boards. This also decouples the builder from the test infrastructure.

Quoted 9 lines

For reference, some discussion on this started in another thread [1], but I'd like to throw this out to a more general audience.

Thanks,
Paul

[1] https://lists.yoctoproject.org/pipermail/yocto/2013-August/017687.html

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#2 lore raw

Hi Darren,

Quoted 15 lines · On Friday 23 August 2013 20:52:41 Darren Hart wrote:
Quoted 8 lines · Paul Eggleton

Automated testing on real hardware is something that a lot of us do already, but the likelihood is that we've all been coming up with our own solutions for this, so it would be really great if we could have a single working solution to cover the generally applicable use cases. As a start though it would be interesting to hear from people who have existing test setups or who are generally interested in this area. I'd like to hear feedback on the following:
* What hardware setup do you use for automated testing?

I am starting to put something together using a customized 19" rack along the lines of those used by OSADL for their real-time farm:

https://www.osadl.org/Test-Rack.test-rack.0.html

Quoted 7 lines

This is a poor man's BladeCenter for embedded boards in that it provides a common interface to each board for the rack infrastructure. The board specific stuff sits on the trays themselves. The trays are easily removable to take a board to a desk for hands-on work.

A smart-switch, web-power strip, and serial concentrator feed into a "gateway server" where services like conmux glue it all together.

All makes sense.

Quoted 10 lines
Quoted 2 lines

* Do you use any software to manage the hardware? Does this include any
existing open source tools?

conmux
dnsmasq
udev (for static usb uart naming)

The testrack has some early-stage tools to support it, but I'm still sorting those out and they aren't particularly robust for initial setup quite yet.

Having a standard way to set up the software side for this kind of infrastructure would certainly be worthwhile I think. One tricky part would perhaps be supporting variations of infrastructure hardware; doing some searches I haven't found a completely definitive open source project for controlling remotely accessible power strips for example, although there are a few scattered projects for doing that with varying levels of recent maintenance.

Quoted 6 lines
Quoted 2 lines

* What would you like to see in a common framework to enable this kind of
testing?

To simplify the board management you allude to above, I believe a pull model, rather than a push, is a better architecture.

Quoted 7 lines

Your builders can complete builds and inject test records into a database with a priority. The boards sit in a boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and push the results back to the database. They select tests based on priority, and the builder doesn't have to do anything at all to manage the boards. This also decouples the builder from the test infrastructure.

Interesting. This would present some challenges with the way the automated tests are currently defined; there's nothing unsolvable that precludes them being run outside of the build environment but they will have to be run somewhere that has python and any required modules installed, presumably an intermediate machine rather than the target machine itself (the latter would not be impossible; but if we did do it that way you couldn't test an image that didn't have python in it.)

We'd also need to come up with a standard definition for the results database and tools to generate reports out of it, but that's something that will likely be needed regardless of the model used for running tests, especially when it comes to recording the results from ptest.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#3 lore raw
Quoted 61 lines · Paul Eggleton

Hi Darren,

Quoted 15 lines · On Friday 23 August 2013 20:52:41 Darren Hart wrote:
Quoted 8 lines · Paul Eggleton

Automated testing on real hardware is something that a lot of us do already, but the likelihood is that we've all been coming up with our own solutions for this, so it would be really great if we could have a single working solution to cover the generally applicable use cases. As a start though it would be interesting to hear from people who have existing test setups or who are generally interested in this area. I'd like to hear feedback on the following:
* What hardware setup do you use for automated testing?

I am starting to put something together using a customized 19" rack along the lines of those used by OSADL for their real-time farm:

https://www.osadl.org/Test-Rack.test-rack.0.html

Quoted 7 lines

This is a poor man's BladeCenter for embedded boards in that it provides a common interface to each board for the rack infrastructure. The board specific stuff sits on the trays themselves. The trays are easily removable to take a board to a desk for hands-on work.

A smart-switch, web-power strip, and serial concentrator feed into a "gateway server" where services like conmux glue it all together.

All makes sense.

Quoted 10 lines
Quoted 2 lines

* Do you use any software to manage the hardware? Does this include any
existing open source tools?

conmux
dnsmasq
udev (for static usb uart naming)

The testrack has some early-stage tools to support it, but I'm still sorting those out and they aren't particularly robust for initial setup quite yet.

Having a standard way to set up the software side for this kind of infrastructure would certainly be worthwhile I think. One tricky part would perhaps be supporting variations of infrastructure hardware; doing some searches I haven't found a completely definitive open source project for controlling remotely accessible power strips for example, although there are a few scattered projects for doing that with varying levels of recent maintenance.

Quoted 6 lines
Quoted 2 lines

* What would you like to see in a common framework to enable this kind of
testing?

To simplify the board management you allude to above, I believe a pull model, rather than a push, is a better architecture.

Quoted 7 lines

Your builders can complete builds and inject test records into a database with a priority. The boards sit in a boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and push the results back to the database. They select tests based on priority, and the builder doesn't have to do anything at all to manage the boards. This also decouples the builder from the test infrastructure.

Quoted 7 lines

Interesting. This would present some challenges with the way the automated tests are currently defined; there's nothing unsolvable that precludes them being run outside of the build environment but they will have to be run somewhere that has python and any required modules installed, presumably an intermediate machine rather than the target machine itself (the latter would not be impossible; but if we did do it that way you couldn't test an image that didn't have python in it.)

A new test-image will need to be created regardless, I think including python is reasonable for the general case. For things like tiny, I suspect we would have a customized test-suite anyway, possibly written in C.

Quoted 5 lines

We'd also need to come up with a standard definition for the results database and tools to generate reports out of it, but that's something that will likely be needed regardless of the model used for running tests, especially when it comes to recording the results from ptest.

Agreed.

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel