New GitHub CI workflow
Hello,
For the last few years we've been relying on my nightly tests to catch any regressions we have in our development tree. The results have been crudely dumped to http://toxicpanda.com and http://toxicpanda.com/performance/. This is a bunch of VM's on a Intel NUC with cronjobs and chewing gum holding it all together. It has worked pretty well, but it is beginning to show it's age, as the overnight runs literally take all night, starting around 8PM my time and finishing 1:30PM my time.
I've upgraded my hardware and re-built the VM setup. The old setup will continue so we have a nice baseline of what's going on, but the new setup will be triggered with GitHub Actions.
# tl;dr
If you are a btrfs developer and are in the https://github.com/btrfs
organization, you can now submit PR's against the "ci" branch in the "linux" repo. This will trigger a full fstests run using my VM's, which currently test 7 different configurations, 5 on x86 and 2 on ARM64. Zoned will be added shortly, I've just been fighting with the thing to get this far so that part is still in the pipeline.
It is highly encouraged that if you've got a decently involved patchset that you take advantage of this system to pre-check your code, either before submitting it to the list for review or in parallel.
# Details
On every run we pull from our "staging" branch in our copy of the "fstests" repo in our https://github.com/btrfs repo. Everybody in the org has commit access to this, if you want new tests run through the CI please do feel free to commit them to this branch and then trigger a CI run.
We also pull down the latest "devel" branch from Sterba's btrfs-progs tree. This will help us catch any output changes or related user space bugs with our fstests runs, as well as allow us to easily test new fs features.
These runs take around 4.5 hours currently. There's a 6 hour timeout, so if there's a deadlock or a panic it'll simply timeout. I haven't figured out a way to capture this information for the results so that's a bummer of a limitation currently.
# Clean runs
In order to make this as easy as possible for us to catch regressions in our own code, my goal is to make sure the baseline runs of fstests run with 0 failures. This means we're utilizing exclude files in this GitHub CI setup. The old setup will continue to run the full suite so we can make sure we're tracking what tests are flakey.
Currently x86 is clean, however ARM still has a few iterations to go before I'll have nailed down all the flakey tests. In the first few weeks please investigate failures to make sure they're not real regressions, or check the existing http://toxicpanda.com list of runs to see if the test already has found to be flakey. In our "staging" branch in our "fstests" repo there is EXCLUDE and EXCLUDE.arm, simply add tests to the relevant file to expunge them from the CI runs.
# We're just ignoring flakey tests?
For now, yes. For things that are easier to reason with, for example the block-tree root feature is incompatible with ~NO_HOLES, patches are preferred to detect and skip the test for specific incompatibilities. However for harder to debug things feel free to simply exclude them. With this list of excludes we can then go back when we have time and nail down each of them and get them fixed up so they're no longer flakey.
# OMFG GITHUB!?!?
Yes, I'm nothing if not lazy, and honestly it took me about a week to work out all the kinks in this particular system. We're already on GH, we already have a lot of processes around using our existing GH org, so I did this with GitHub actions. I have 0 interest in debating this particular choice. If you aren't a core developer you aren't my target audience. If you don't want to use the CI you don't have to, this is simply a convenience offered to those of us who do this for our job.
# I'm not in the GH Btrfs org/don't have a GH account and want to use the CI?
If you have a sufficiently large patchset you would like run against the CI for you all you have to do is ask one of the Btrfs developers and we'd be happy to push a PR for you so you can get your patches tested.
If you would like to be added to the GH Btrfs Org and aren't a current member please email myself or Dave Sterba and we can decide if you can be added as a member, but this will be limited to consistent contributors. We are always happy to submit PR's on your behalf if you're just looking for some extra validation of a large contribution.
Thanks,
Josef
Quoted 85 lines · Josef Bacik
Hello,
For the last few years we've been relying on my nightly tests to catch any regressions we have in our development tree. The results have been crudely dumped to http://toxicpanda.com and http://toxicpanda.com/performance/. This is a bunch of VM's on a Intel NUC with cronjobs and chewing gum holding it all together. It has worked pretty well, but it is beginning to show it's age, as the overnight runs literally take all night, starting around 8PM my time and finishing 1:30PM my time.
I've upgraded my hardware and re-built the VM setup. The old setup will continue so we have a nice baseline of what's going on, but the new setup will be triggered with GitHub Actions.
# tl;dr
If you are a btrfs developer and are in the https://github.com/btrfs
organization, you can now submit PR's against the "ci" branch in the "linux" repo. This will trigger a full fstests run using my VM's, which currently test 7 different configurations, 5 on x86 and 2 on ARM64. Zoned will be added shortly, I've just been fighting with the thing to get this far so that part is still in the pipeline.It is highly encouraged that if you've got a decently involved patchset that you take advantage of this system to pre-check your code, either before submitting it to the list for review or in parallel.
# Details
On every run we pull from our "staging" branch in our copy of the "fstests" repo in our https://github.com/btrfs repo. Everybody in the org has commit access to this, if you want new tests run through the CI please do feel free to commit them to this branch and then trigger a CI run.
We also pull down the latest "devel" branch from Sterba's btrfs-progs tree. This will help us catch any output changes or related user space bugs with our fstests runs, as well as allow us to easily test new fs features.
These runs take around 4.5 hours currently. There's a 6 hour timeout, so if there's a deadlock or a panic it'll simply timeout. I haven't figured out a way to capture this information for the results so that's a bummer of a limitation currently.
# Clean runs
In order to make this as easy as possible for us to catch regressions in our own code, my goal is to make sure the baseline runs of fstests run with 0 failures. This means we're utilizing exclude files in this GitHub CI setup. The old setup will continue to run the full suite so we can make sure we're tracking what tests are flakey.
Currently x86 is clean, however ARM still has a few iterations to go before I'll have nailed down all the flakey tests. In the first few weeks please investigate failures to make sure they're not real regressions, or check the existing http://toxicpanda.com list of runs to see if the test already has found to be flakey. In our "staging" branch in our "fstests" repo there is EXCLUDE and EXCLUDE.arm, simply add tests to the relevant file to expunge them from the CI runs.
# We're just ignoring flakey tests?
For now, yes. For things that are easier to reason with, for example the block-tree root feature is incompatible with ~NO_HOLES, patches are preferred to detect and skip the test for specific incompatibilities. However for harder to debug things feel free to simply exclude them. With this list of excludes we can then go back when we have time and nail down each of them and get them fixed up so they're no longer flakey.
# OMFG GITHUB!?!?
Yes, I'm nothing if not lazy, and honestly it took me about a week to work out all the kinks in this particular system. We're already on GH, we already have a lot of processes around using our existing GH org, so I did this with GitHub actions. I have 0 interest in debating this particular choice. If you aren't a core developer you aren't my target audience. If you don't want to use the CI you don't have to, this is simply a convenience offered to those of us who do this for our job.
# I'm not in the GH Btrfs org/don't have a GH account and want to use the CI?
If you have a sufficiently large patchset you would like run against the CI for you all you have to do is ask one of the Btrfs developers and we'd be happy to push a PR for you so you can get your patches tested.
Quoted 6 lines
If you would like to be added to the GH Btrfs Org and aren't a current member please email myself or Dave Sterba and we can decide if you can be added as a member, but this will be limited to consistent contributors. We are always happy to submit PR's on your behalf if you're just looking for some extra validation of a large contribution.
Will we move btrfs-progs to this GitHub organization?
Signature · 1 line
真実はいつも一つ!/ Always, there's only one truth!
Quoted 23 lines · Josef Bacik
Hello,
For the last few years we've been relying on my nightly tests to catch any regressions we have in our development tree. The results have been crudely dumped to http://toxicpanda.com and http://toxicpanda.com/performance/. This is a bunch of VM's on a Intel NUC with cronjobs and chewing gum holding it all together. It has worked pretty well, but it is beginning to show it's age, as the overnight runs literally take all night, starting around 8PM my time and finishing 1:30PM my time.
I've upgraded my hardware and re-built the VM setup. The old setup will continue so we have a nice baseline of what's going on, but the new setup will be triggered with GitHub Actions.
# tl;dr
If you are a btrfs developer and are in the https://github.com/btrfs
organization, you can now submit PR's against the "ci" branch in the "linux" repo. This will trigger a full fstests run using my VM's, which currently test 7 different configurations, 5 on x86 and 2 on ARM64. Zoned will be added shortly, I've just been fighting with the thing to get this far so that part is still in the pipeline.
Quoted 3 lines
It is highly encouraged that if you've got a decently involved patchset that you take advantage of this system to pre-check your code, either before submitting it to the list for review or in parallel.
I would recommend to use the CI in addition to any local tests, this should not be a substitute. All of us have a different configurations of VMs, machines or .config and this has proven to be very useful.
Quoted 9 lines
# OMFG GITHUB!?!?
Yes, I'm nothing if not lazy, and honestly it took me about a week to work out all the kinks in this particular system. We're already on GH, we already have a lot of processes around using our existing GH org, so I did this with GitHub actions. I have 0 interest in debating this particular choice. If you aren't a core developer you aren't my target audience. If you don't want to use the CI you don't have to, this is simply a convenience offered to those of us who do this for our job.
The GH actions and configuration are partially files tracked in git, partially doen only on the web (permissions, integration with other services), it would be good to have it documented somewhere. I put notes to changelogs when enabling something etc, or we can have a separate document that enumerates what needs to be clicked and where. I'm not expecting GH to disappear anytime soon. From past experience the travis-ci hosted CI for btrfs-progs became unusuable and none of the scripts could have been used directly but at least it documented which tests were run, what packages are needed to install etc. Such changes happen rarely but it is useful to start with something than to spend an extra week figuring everything out again.
Quoted 10 lines · Neal Gompa
Quoted 9 lines · Josef Bacik
you don't have to, this is simply a convenience offered to those of us who do this for our job.
# I'm not in the GH Btrfs org/don't have a GH account and want to use the CI?
If you have a sufficiently large patchset you would like run against the CI for you all you have to do is ask one of the Btrfs developers and we'd be happy to push a PR for you so you can get your patches tested.
Quoted 8 lines
Quoted 6 lines
If you would like to be added to the GH Btrfs Org and aren't a current member please email myself or Dave Sterba and we can decide if you can be added as a member, but this will be limited to consistent contributors. We are always happy to submit PR's on your behalf if you're just looking for some extra validation of a large contribution.
Will we move btrfs-progs to this GitHub organization?
Why? I can create a repository and push the releases there, but for development a lot of things would have to be changed to make everything work, scripts, integration, permissions, etc.