Autobuilder changes
I've take this quiet period as an opportunity to make some significant autobuilder changes:
The targets have all been renamed and split up. "nightly-arm" became "qemuarm" and "beaglebone". The "nightly-" prefixes were dropped.
The nightly target was dropped and replaced by "a-quick" and "a-full". The difference are a number of targets which don't often find problems. We're also ramping down on the amount of testing qemumips-lsb/qemuppc- lsb get for example as they're more minor architectures as things stand. The "a-" prefix is for sorting in the UI and to separate them out from the mass of other builds.
There are now selftest targets per "distro" which can replace the testing that Intel QA have been doing. This is only done for "a-full" and the selftest on a distro at random is done by "a-quick".
Buildhistory is only being generated for the qemu* targets.
The "template" used for testing qemu or "real-hardware" targets was standardised as there were weird differences for now good reason.
The structure of "non-release" builds on
https://autobuilder.yocto.io/pub/ was cleaned up and now appears under a single date+number under the non-release directory which was how it was meant to work. All the other spurious output that was there has been cleaned up.
XML test results are now being published for all builds, e.g.:
https://autobuilder.yocto.io/pub/non-release/20181122-11/qemuarm64/
Ptest was added to "a-full" builds for arm and x86 although the arm builds currently time out. The plan will be to enable arm when we have an arm build server with KVM support in our rack (next week?). ptest execution for x86 takes around 3 hours and some sample result output is:
https://autobuilder.yocto.io/pub/non-release/20181122-11/qemux86-64-ptest/
"meta-oe" and "meta-virt" test targets were added for experimentation, testing meta-openembedded and meta-virtualization respectively. These are work in progress currently just doing a "world -k" build for qemux86-64 which fails (they take around 2.5 hours).
We now run "a-quick" on master at 1am every day which will make buildhistory more up to date and useful for the master branch and give us better comparison data for -next and mut.
wine is installed onto the ubuntu1804 workers and the mingw testing was split into a meta-mingw target, now with 32 and 64 bit build tests and 64 bit runtime tests. With master-next of meta-mingw this appears to run the test. Huge kudos to Joshua Watt for putting that testing together. If we install more onto the workers we could probably test 32 bit as well, I'm torn on that right now as its a lot more dependencies on the workers.
Building older releases is now broken until I sort out the corresponding helper changes for those branches but one thing at a time!
If anyone would like to help with xml results file analysis tooling, that is where we need work next to be able to summarise the ptest results and do results comparison. I know Ee Peng has some work in this area which I will be looking at next.
Cheers,
Richard
Quoted 2 lines · Richard Purdie
I've take this quiet period as an opportunity to make some significant autobuilder changes:
I knew I was missing something:
Successful builds are now deleted when they complete using the background deletion (clobberdir). This should reduce the clutter lying around on the worker filesystems. It also means builds shouldn't be contending with background deletion as much.
Failed builds have their build directory renamed to "build-renamed". This means that the paths all become invalid and can't be accidentally referenced from other builds.
I did a sweep through the workers deleting all the old build directories and cleared a lot of the diskspace on them.
Cheers,
Richard
Quoted 37 lines · Richard Purdie
I've take this quiet period as an opportunity to make some significant autobuilder changes:
The targets have all been renamed and split up. "nightly-arm" became "qemuarm" and "beaglebone". The "nightly-" prefixes were dropped.
The nightly target was dropped and replaced by "a-quick" and "a-full". The difference are a number of targets which don't often find problems. We're also ramping down on the amount of testing qemumips-lsb/qemuppc- lsb get for example as they're more minor architectures as things stand. The "a-" prefix is for sorting in the UI and to separate them out from the mass of other builds.
There are now selftest targets per "distro" which can replace the testing that Intel QA have been doing. This is only done for "a-full" and the selftest on a distro at random is done by "a-quick".
Buildhistory is only being generated for the qemu* targets.
The "template" used for testing qemu or "real-hardware" targets was standardised as there were weird differences for now good reason.
The structure of "non-release" builds on
https://autobuilder.yocto.io/pub/ was cleaned up and now appears under a single date+number under the non-release directory which was how it was meant to work. All the other spurious output that was there has been cleaned up.XML test results are now being published for all builds, e.g.:
https://autobuilder.yocto.io/pub/non-release/20181122-11/qemuarm64/Ptest was added to "a-full" builds for arm and x86 although the arm builds currently time out. The plan will be to enable arm when we have an arm build server with KVM support in our rack (next week?). ptest execution for x86 takes around 3 hours and some sample result output is:
Quoted 8 lines
https://autobuilder.yocto.io/pub/non-release/20181122-11/qemux86-64-ptest/
"meta-oe" and "meta-virt" test targets were added for experimentation, testing meta-openembedded and meta-virtualization respectively. These are work in progress currently just doing a "world -k" build for qemux86-64 which fails (they take around 2.5 hours).
We now run "a-quick" on master at 1am every day which will make
What Time zone for 1am?
Quoted 11 lines
buildhistory more up to date and useful for the master branch and give us better comparison data for -next and mut.
wine is installed onto the ubuntu1804 workers and the mingw testing was split into a meta-mingw target, now with 32 and 64 bit build tests and 64 bit runtime tests. With master-next of meta-mingw this appears to run the test. Huge kudos to Joshua Watt for putting that testing together. If we install more onto the workers we could probably test 32 bit as well, I'm torn on that right now as its a lot more dependencies on the workers.
Quoted 3 lines
Building older releases is now broken until I sort out the corresponding helper changes for those branches but one thing at a time!
Does that include Thud ? I expect sumo and below to be problems.
Do the changes need to roll from newest to oldest or are they independent? I was hoping to get Sumo tested using the original QA process to buy us time to get the automation working on it. Does this changes that ?
- armin
Quoted 12 lines
If anyone would like to help with xml results file analysis tooling, that is where we need work next to be able to summarise the ptest results and do results comparison. I know Ee Peng has some work in this area which I will be looking at next.
Cheers,
Richard
Quoted 4 lines · akuster
Quoted 1 line · Richard Purdie
We now run "a-quick" on master at 1am every day which will make
What Time zone for 1am?
I'm not 100% sure but I think its UTC.
Quoted 10 lines
Quoted 3 lines
Building older releases is now broken until I sort out the corresponding helper changes for those branches but one thing at a time!
Does that include Thud ? I expect sumo and below to be problems.
Do the changes need to roll from newest to oldest or are they independent? I was hoping to get Sumo tested using the original QA process to buy us time to get the automation working on it. Does this changes that ?
Good news is I've fixed the older releases now and run test builds to check things. There were a couple of fixes needed but I've sorted those and done a test artefact deployment too. This just updates everything to use the new builder names.
The QA process is unchanged. If a given build generates a test report, whichever release it is, it will be collected up but if not that is also fine. This means we can opt in to the new code if/as/when we backport anything. So we're good for sumo.
We do need to sort these gpg issues across various stable releases though and probably tweak master too.
There were some test results ordering changes I'd be tempted to backport just as they make reading the build logs so much easier...
Cheers,
Richard