U-Boot Regression Testing
Hello,
I have some questions about how U-Boot regression testing works. I am assuming some regression testing happens during the release period across some representative sample of boards and architectures.
I know people check for compilation failures with MAKEALL, but I am wondering about runtime testing. I also understand that testing platform specific boot code is rather difficult (or easy depending on your perspective) - it boots or it does not boot.
To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?
Googling did not turn up much on how this is done.
On the social side -- is that something the community helps out with or something DENX does, or a mix?
Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)
This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?
We have a project, of which U-Boot is a part, that is starting to span multiple boards and architectures. We make a few modifications to U-Boot and I want to start automated regression testing as the number of boards increases.
If an existing framework exists that folks are happy with I would love to hear about it. Equally, I am interested to hear about what did *not* work for people. Whatever method our project uses the scripts will be publicly available.
Cheers,
Curt
Hi Curt,
Quoted 32 lines · Curt Brune
Hello,
I have some questions about how U-Boot regression testing works. I am assuming some regression testing happens during the release period across some representative sample of boards and architectures.
I know people check for compilation failures with MAKEALL, but I am wondering about runtime testing. I also understand that testing platform specific boot code is rather difficult (or easy depending on your perspective) - it boots or it does not boot.
To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?
Googling did not turn up much on how this is done.
On the social side -- is that something the community helps out with or something DENX does, or a mix?
Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)
This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?
We have a project, of which U-Boot is a part, that is starting to span multiple boards and architectures. We make a few modifications to U-Boot and I want to start automated regression testing as the number of boards increases.
Quoted 4 lines
If an existing framework exists that folks are happy with I would love to hear about it. Equally, I am interested to hear about what did *not* work for people. Whatever method our project uses the scripts will be publicly available.
You can use sandbox, which is a build of U-Boot that runs under Linux. There are some existing tests in test/ - you mention FIT support and there is a basic test for that (could be expanded, in fact I may have a patch somewhere).
Note that there is also buildman which aims to be the new MAKEALL eventually.
Regards,
Simon
Quoted 7 lines
Cheers,
Curt
_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot
Dear Curt Brune,
Quoted 4 lines · In message <[email protected]> you wrote:
To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?
Have a look at DUTS [1]; the test cases implemented there serve a double purpose: 1) they help testing (some of) the functionality of U-Boot (and Linux) on a specific board, and 2) they generate the needed include files to produce a board-specific version of the DULG
[2]
[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] http://www.denx.de/wiki/DULG/BoardSelect
Quoted 2 lines
On the social side -- is that something the community helps out with or something DENX does, or a mix?
Yes, of course this is mainly a community effort. We at DENX have only a limited number of boards in our lab (just 138 at the moment), so all board maintainers are appealed for testing on their hardware, and al custodians will test on a set of boards they have access to.
Quoted 2 lines
Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)
Here at DENX we use DUTS, though not as much as we should do ...
Quoted 3 lines
This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?
DUTS is based on expect (and thus tcl, sigh!)
Best regards,
Wolfgang Denk
Signature · 4 lines
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de "Free markets select for winning solutions." - Eric S. Raymond
----- Original Message -----
Quoted 8 lines
Have a look at DUTS [1]; the test cases implemented there serve a double purpose: 1) they help testing (some of) the functionality of U-Boot (and Linux) on a specific board, and 2) they generate the needed include files to produce a board-specific version of the DULG
[2][1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] http://www.denx.de/wiki/DULG/BoardSelect
Thanks. That was the type of info I was curious about. Very helpful.
Cheers,
Curt
Hi Curt,
Sorry for late reply.
Quoted 5 lines
Hello,
I have some questions about how U-Boot regression testing works. I am assuming some regression testing happens during the release period across some representative sample of boards and architectures.
I can only present my approach - I test the board (TRATS) often before the official u-boot release.
I don't have any list of commands to test -> I just test the ones often used (mm) or the one to which I contribute (pmic, dfu, ums).
Quoted 9 lines
I know people check for compilation failures with MAKEALL, but I am wondering about runtime testing. I also understand that testing platform specific boot code is rather difficult (or easy depending on your perspective) - it boots or it does not boot.
To be concrete -- how are core U-Boot commands and features tested? For example how do you test that FIT image support is not broken or that the 'env' command and all its options work properly?
As with DFU or UMS - I have got an automated test script for HOST PC to test if data (including corner cases data size) is written and read correctly from the eMMC medium. However those tests require typing "dfu mmc 0" or "ums 0" commands on the target.
This posts reminds me that I shall post them....
Quoted 8 lines
Googling did not turn up much on how this is done.
On the social side -- is that something the community helps out with or something DENX does, or a mix?
Are you using a test framework of some kind, either home grown or open source? These things tend to become home grown over time :)
I think that such scripts shall be added to ./test directory.
Quoted 4 lines
This kind of testing usually takes the form of 'chat' scripts communicating over serial consoles. Perhaps you are using expect, pexpect, python nose?
Unfortunately I don't use serial console. I even thought of writing a special USB gadget (or extent DFU) to provide a convenient way to test if e.g. eMMC write works:
- Read file from FS via e.g. ext4load
- Calculate CRC (crc)
- Store the file with other name
- Read it
- Calculate CRC and compare
I'm also curious how other perform their tests. Since I _really_ want to avoid over-engineered solution. Maybe someone do it better, simpler?
Quoted 5 lines
We have a project, of which U-Boot is a part, that is starting to span multiple boards and architectures. We make a few modifications to U-Boot and I want to start automated regression testing as the number of boards increases.
As I have written above. The best approach would be to use only board under test and host PC.
However it might be necessary to use the uC based "interface" board [1] connected to HOST PC to perform power cycle (when we cannot use u-boot's "reset" command).
Quoted 3 lines
If an existing framework exists that folks are happy with I would love to hear about it.
+1 here
Quoted 2 lines
Equally, I am interested to hear about what did
*not* work for people.
I would like to avoid using the [1]. And night run tests would be also very welcome.
Quoted 5 lines
Whatever method our project uses the scripts
will be publicly available.Cheers,
Curt
Signature · 5 lines
Best regards, Lukasz Majewski Samsung R&D Institute Poland (SRPOL) | Linux Platform Group