Skip to contentSkip to conversation

U-Boot Regression Testing

U-Boot Curt Brune 4 participants 5 messages Oct 9, 2013 – Oct 11, 2013

lore
Share & tools
lore

Curt Brune

author
#0 lore raw

Hello,

I have some questions about how U-Boot regression testing works. I am assuming some regression testing happens during the release period across some representative sample of boards and architectures.

I know people check for compilation failures with MAKEALL, but I am wondering about runtime testing. I also understand that testing platform specific boot code is rather difficult (or easy depending on your perspective) - it boots or it does not boot.

To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?

Googling did not turn up much on how this is done.

On the social side -- is that something the community helps out with or something DENX does, or a mix?

Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)

This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?

We have a project, of which U-Boot is a part, that is starting to span multiple boards and architectures. We make a few modifications to U-Boot and I want to start automated regression testing as the number of boards increases.

If an existing framework exists that folks are happy with I would love to hear about it. Equally, I am interested to hear about what did *not* work for people. Whatever method our project uses the scripts will be publicly available.

Cheers,
Curt

#1 lore raw

Hi Curt,

Quoted 32 lines · Curt Brune

Hello,

I have some questions about how U-Boot regression testing works. I am assuming some regression testing happens during the release period across some representative sample of boards and architectures.

I know people check for compilation failures with MAKEALL, but I am wondering about runtime testing. I also understand that testing platform specific boot code is rather difficult (or easy depending on your perspective) - it boots or it does not boot.

To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?

Googling did not turn up much on how this is done.

On the social side -- is that something the community helps out with or something DENX does, or a mix?

Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)

This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?

We have a project, of which U-Boot is a part, that is starting to span multiple boards and architectures. We make a few modifications to U-Boot and I want to start automated regression testing as the number of boards increases.

Quoted 4 lines

If an existing framework exists that folks are happy with I would love to hear about it. Equally, I am interested to hear about what did *not* work for people. Whatever method our project uses the scripts will be publicly available.

You can use sandbox, which is a build of U-Boot that runs under Linux. There are some existing tests in test/ - you mention FIT support and there is a basic test for that (could be expanded, in fact I may have a patch somewhere).

Note that there is also buildman which aims to be the new MAKEALL eventually.

Regards,
Simon

Quoted 7 lines

Cheers,
Curt
_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

#2 lore raw
In reply to Curt Brune

Dear Curt Brune,

Quoted 4 lines · In message <[email protected]> you wrote:

To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?

Have a look at DUTS [1]; the test cases implemented there serve a double purpose: 1) they help testing (some of) the functionality of U-Boot (and Linux) on a specific board, and 2) they generate the needed include files to produce a board-specific version of the DULG
[2]

[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] http://www.denx.de/wiki/DULG/BoardSelect

Quoted 2 lines

On the social side -- is that something the community helps out with or something DENX does, or a mix?

Yes, of course this is mainly a community effort. We at DENX have only a limited number of boards in our lab (just 138 at the moment), so all board maintainers are appealed for testing on their hardware, and al custodians will test on a set of boards they have access to.

Quoted 2 lines

Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)

Here at DENX we use DUTS, though not as much as we should do ...

Quoted 3 lines

This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?

DUTS is based on expect (and thus tcl, sigh!)

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"Free markets select for winning solutions."        - Eric S. Raymond
author
#3 lore raw

----- Original Message -----

Quoted 8 lines

Have a look at DUTS [1]; the test cases implemented there serve a double purpose: 1) they help testing (some of) the functionality of U-Boot (and Linux) on a specific board, and 2) they generate the needed include files to produce a board-specific version of the DULG
[2]

[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] http://www.denx.de/wiki/DULG/BoardSelect

Thanks. That was the type of info I was curious about. Very helpful.

Cheers,
Curt

#4 lore raw
In reply to Curt Brune

Hi Curt,

Sorry for late reply.

Quoted 5 lines

Hello,

I have some questions about how U-Boot regression testing works. I am assuming some regression testing happens during the release period across some representative sample of boards and architectures.

I can only present my approach - I test the board (TRATS) often before the official u-boot release.

I don't have any list of commands to test -> I just test the ones often used (mm) or the one to which I contribute (pmic, dfu, ums).

Quoted 9 lines

I know people check for compilation failures with MAKEALL, but I am wondering about runtime testing. I also understand that testing platform specific boot code is rather difficult (or easy depending on your perspective) - it boots or it does not boot.

To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?

As with DFU or UMS - I have got an automated test script for HOST PC to test if data (including corner cases data size) is written and read correctly from the eMMC medium. However those tests require typing "dfu mmc 0" or "ums 0" commands on the target.

This posts reminds me that I shall post them....

Quoted 8 lines

Googling did not turn up much on how this is done.

On the social side -- is that something the community helps out with or something DENX does, or a mix?

Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)

I think that such scripts shall be added to ./test directory.

Quoted 4 lines

This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?

Unfortunately I don't use serial console. I even thought of writing a special USB gadget (or extent DFU) to provide a convenient way to test if e.g. eMMC write works:

- Read file from FS via e.g. ext4load
- Calculate CRC (crc)
- Store the file with other name
- Read it
- Calculate CRC and compare

I'm also curious how other perform their tests. Since I _really_ want to avoid over-engineered solution. Maybe someone do it better, simpler?

Quoted 5 lines

We have a project, of which U-Boot is a part, that is starting to span multiple boards and architectures. We make a few modifications to U-Boot and I want to start automated regression testing as the number of boards increases.

As I have written above. The best approach would be to use only board under test and host PC.
However it might be necessary to use the uC based "interface" board [1] connected to HOST PC to perform power cycle (when we cannot use u-boot's "reset" command).

Quoted 3 lines

If an existing framework exists that folks are happy with I would love to hear about it.

+1 here

Quoted 2 lines

Equally, I am interested to hear about what did
*not* work for people.

I would like to avoid using the [1]. And night run tests would be also very welcome.

Quoted 5 lines

Whatever method our project uses the scripts
will be publicly available.

Cheers,
Curt

Signature · 5 lines
Best regards,

Lukasz Majewski

Samsung R&D Institute Poland (SRPOL) | Linux Platform Group