Skip to contentSkip to conversation

XFS LTS backport cabal

DISCUSSION Darrick J. Wong 6 participants 7 messages May 25, 2022 – May 26, 2022

lore
Share & tools
lore
#0 lore raw

Hi everyone,

As most of the people cc'd on this message are aware, there's been a recent groundswell of interest in increasing the amount of staff time assigned to backporting bug fixes to LTS kernels. As part of preparing to resume maintainership on June 5th, I thought it would be a good idea to introduce all the participants (that I know of!) and to capture a summary of everyone's thoughts w.r.t. how to make stable backports happen.

First, some introductions: Leah and Ted work at Google, and they've expressed interest in 5.15 LTS backports. Dave and Eric are the other upstream XFS maintainers, so I've included them. Amir seems to be working on 5.10 LTS backports for his employer(?). Chandan and I work at Oracle (obviously) and he's also been working on a few 5.15 LTS backports.

I won't speak for other organizations, but we (Oracle) are also interested in stable backports for the 5.4 and 4.14 LTS kernels, since we have customers running <cough> derivatives of those kernels. Given what I've heard from others, many kernel distributors lean on the LTS kernels.

The goal of this thread, then, is to shed some light on who's currently doing what to reduce duplication of LTS work, and to make sure that we're all more or less on the same page with regards to what we will and won't try to push to stable. (A side goal of mine is to help everyone working on the stable branches to avoid the wrath and unhelpful form letters of the stable maintainers.)

Briefly, I think the patches that flow into XFS could be put into three rough categories:

(a) Straightforward fixes. These are usually pretty simple fixes (e.g. omitted errno checking, insufficient validation, etc.) sometimes get proper Fixes tags, which means that AUTOSEL can be of some benefit.

(b) Probable fixes. Often these aren't all that obvious -- for example, the author may be convinced that they correct a mis-interaction between subsystems, but we would like the changes to soak in upstream for a few months to build confidence that they solve the problem and without causing more problems.

(c) Everything else. New features, giant refactorings, etc. These generally should not be backported, unless someone has a /really/ good reason.

Here are a few principles I'd like to see guiding stable backport efforts:

1. AUTOSEL is a good tool to _start_ the process of identifying low hanging fruit to backport. Automation is our friend, but XFS is complex so we still need people who have kept up with linux-xfs to know what's appropriate (and what compile tests can't find) to finish the process.

2. Some other tag for patches that could be a fix, but need a few months to soak. This is targetted at (b), since I'm terrible at remembering that there are patches that are reaching ripeness.

3. fstesting -- new patches proposed for stable branches shouldn't introduce new regressions, and ideally there would also be a regression test that would now pass. As Dave and I have stated in the past, fstests is a big umbrella of a test suite, which implies that A/B testing is the way to go. I think at least Zorro and I would like to improve the tagging in fstests to make it more obvious which tests contain enough randomness that they cannot be expected to behave 100% reliably.

Here's a couple of antipatterns from the past:

i. Robots shovelling patches into stable kernels with no testing.

ii. Massively large backports. New features don't go to stable kernels, and I doubt the stable kernel maintainers will accept that anyway. I grok the temptation to backport more so that it's easier to land future fixes via AUTOSEL, but I personally wouldn't endorse frontloading a bunch of work to chase a promise of less future work.

And a question or two:

a> I've been following the recent fstests threads, and it seems to me that there are really two classes of users -- sustaining people who want fstests to run reliably so they can tell if their backports have broken anything; and developers, who want the randomness to try to poke into dusty corners of the filesystem. Can we make it easier to associate random bits of data (reliability rates, etc.) with a given fstests configuration? And create a test group^Wtag for the tests that rely on RNGs to shake things up?

b> Testing relies very heavily on being able to spin up a lot of testing resources. Can/should we make it easier for people with a kernel.org account to get free(ish) cloud accounts with the LF members who are also cloud vendors?

Thoughts? Flames?

--D

#1 lore raw
Quoted 3 lines · Darrick J. Wong

Hi everyone,

Hi Darrick!

Thanks for making this introduction.

I've added Tyler (Microsoft) for his interest in 5.10 LTS backports and Luis (Samsung) who has worked with the stable maintainers on xfs backports in the past and is collaborating with me on 5.10 LTS testing these days.

Quoted 8 lines

As most of the people cc'd on this message are aware, there's been a recent groundswell of interest in increasing the amount of staff time assigned to backporting bug fixes to LTS kernels. As part of preparing to resume maintainership on June 5th, I thought it would be a good idea to introduce all the participants (that I know of!) and to capture a summary of everyone's thoughts w.r.t. how to make stable backports happen.

Quoted 4 lines

First, some introductions: Leah and Ted work at Google, and they've expressed interest in 5.15 LTS backports. Dave and Eric are the other upstream XFS maintainers, so I've included them. Amir seems to be working on 5.10 LTS backports for his employer(?).

Correct, my goals are twofold:

1. Consolidate duplicate efforts on stable kernel -
For CTERA products that are shippened as a VM image, I do not *need* my backports to CTERA kernel to go to LTS. I *want* them to go to LTS. As a community of developers who work for different companies, we work so well together on upstream and so poorly apart on the stable kernels that we each maintain.
I would like to see that change.

2. For CTERA products that run in containers in the cloud, we rely on the flow of fixes to LTS kernels that will someday flow into the container host OS of the cloud provider. Unless I start sending patches to cloud providers directly,
LTS is my only path to get things fixed in the future.
For this use case, we do not even have the luxury of assuming a recent LTS kernel. The major cloud providers released host OS images based on 5.10 only late 2021.

Quoted 34 lines

Chandan and I work
at Oracle (obviously) and he's also been working on a few 5.15 LTS backports.

I won't speak for other organizations, but we (Oracle) are also interested in stable backports for the 5.4 and 4.14 LTS kernels, since we have customers running <cough> derivatives of those kernels. Given what I've heard from others, many kernel distributors lean on the LTS kernels.

The goal of this thread, then, is to shed some light on who's currently doing what to reduce duplication of LTS work, and to make sure that we're all more or less on the same page with regards to what we will and won't try to push to stable. (A side goal of mine is to help everyone working on the stable branches to avoid the wrath and unhelpful form letters of the stable maintainers.)

Briefly, I think the patches that flow into XFS could be put into three rough categories:

(a) Straightforward fixes. These are usually pretty simple fixes (e.g. omitted errno checking, insufficient validation, etc.) sometimes get proper Fixes tags, which means that AUTOSEL can be of some benefit.

(b) Probable fixes. Often these aren't all that obvious -- for example, the author may be convinced that they correct a mis-interaction between subsystems, but we would like the changes to soak in upstream for a few months to build confidence that they solve the problem and without causing more problems.

(c) Everything else. New features, giant refactorings, etc. These generally should not be backported, unless someone has a /really/ good reason.

Quoted 8 lines

Here are a few principles I'd like to see guiding stable backport efforts:

1. AUTOSEL is a good tool to _start_ the process of identifying low hanging fruit to backport. Automation is our friend, but XFS is complex so we still need people who have kept up with linux-xfs to know what's appropriate (and what compile tests can't find) to finish the process.

I am very happy that you stepped up to write these expectations. Please spell out the process you want to see, because it was not clear to Ted and to me how we should post the xfs stable candidates.

Should we post to xfs list and wait for explicit ACK/RVB on every patch? Should we post to xfs list and if no objections are raised post to stable? For those of you not subscribed to xfs list, I have just sent my first crop of 5.10 candidates yesterday [1]. Still waiting to see how that plays out.

[1] https://lore.kernel.org/linux-xfs/[email protected]/

Quoted 3 lines

2. Some other tag for patches that could be a fix, but need a few months to soak. This is targetted at (b), since I'm terrible at remembering that there are patches that are reaching ripeness.

The question is, to soak in 'master' or to soak in a release? which means to soak in the stable kernel.

My opinion is that if we want exposure to real users that use diverse userspace tools, then soaking in stable is the right answer, so typically
(b) patches would be soaking ~2 months in -rc and then ~2 more months in .y and at that point they can ONLY be considered for LTS, because that .y release (which is not LTS) is EOL.

Quoted 22 lines

3. fstesting -- new patches proposed for stable branches shouldn't introduce new regressions, and ideally there would also be a regression test that would now pass. As Dave and I have stated in the past, fstests is a big umbrella of a test suite, which implies that A/B testing is the way to go. I think at least Zorro and I would like to improve the tagging in fstests to make it more obvious which tests contain enough randomness that they cannot be expected to behave 100% reliably.

Here's a couple of antipatterns from the past:

i. Robots shovelling patches into stable kernels with no testing.

ii. Massively large backports. New features don't go to stable kernels, and I doubt the stable kernel maintainers will accept that anyway. I grok the temptation to backport more so that it's easier to land future fixes via AUTOSEL, but I personally wouldn't endorse frontloading a bunch of work to chase a promise of less future work.

And a question or two:

Quoted 9 lines

a> I've been following the recent fstests threads, and it seems to me that there are really two classes of users -- sustaining people who want fstests to run reliably so they can tell if their backports have broken anything; and developers, who want the randomness to try to poke into dusty corners of the filesystem. Can we make it easier to associate random bits of data (reliability rates, etc.) with a given fstests configuration? And create a test group^Wtag for the tests that rely on RNGs to shake things up?

I'm a big fan of that idea :)

Quoted 7 lines

b> Testing relies very heavily on being able to spin up a lot of testing resources. Can/should we make it easier for people with a kernel.org account to get free(ish) cloud accounts with the LF members who are also cloud vendors?

Thoughts? Flames?

Here is one thought - the stable candidate review process is bound to add more work to the overloaded xfs maintainer.

I think it is a classic role to delegate to xfs stable tree maintainer(s). The xfs stable maintainer(s) job would be to ping reviewers for acks make sure patches are tested according to the standard and then curate a tree and communicate with the stable maintainers.

Ideally, there will be a maintainer per LTS tree, which could naturally align with the interest of employers and easier for developers to justify spent time, but the load can be shared among maintainers, as nagging for review is often not LTS specific.

If xfs maintainers are willing to put their trust in me, I volunteer for the job of xfs stable patch hurding in general and/or 5.10 LTS specific, which I am also contributing to.

Thanks,
Amir.

#2 lore raw
In reply to Darrick J. Wong
Quoted 10 lines · Darrick J. Wong

Hi everyone,

3. fstesting -- new patches proposed for stable branches shouldn't introduce new regressions, and ideally there would also be a regression test that would now pass. As Dave and I have stated in the past, fstests is a big umbrella of a test suite, which implies that A/B testing is the way to go. I think at least Zorro and I would like to improve the tagging in fstests to make it more obvious which tests contain enough randomness that they cannot be expected to behave 100% reliably.

It would be nice to find an agreement on testing requirements. I have attached some ideas on configs/number of tests/etc as well as the status of my work on 5.15 below.

Quoted 8 lines

a> I've been following the recent fstests threads, and it seems to me that there are really two classes of users -- sustaining people who want fstests to run reliably so they can tell if their backports have broken anything; and developers, who want the randomness to try to poke into dusty corners of the filesystem. Can we make it easier to associate random bits of data (reliability rates, etc.) with a given fstests configuration? And create a test group^Wtag for the tests that rely on RNGs to shake things up?

This would be great!

Quoted 5 lines

Thoughts? Flames?

--D

This thread had good timing :) I have been working on setting up some automated testing. Currently, 5.15.y is our priority so I have started working on this branch.

Patches are being selected by simply searching for the “Fixes” tag and applying if the commit-to-be-fixed is in the stable branch, but AUTOSEL would be nice, so I’ll start playing around with that. Amir, it would be nice to sync up the patch selection process. I can help share the load, especially for 5.15.

Selecting just the tagged “Fixes” for 5.15.y for patches through 5.17.2, 15 patches were found and applied - if there are no complaints about the testing setup, I can go ahead and send out this batch:

c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects 5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname 1090427bf18f xfs: remove xfs_inew_wait
089558bc7ba7 xfs: remove all COW fork extents when remounting readonly 7993f1a431bc xfs: only run COW extent recovery when there are no live extents 09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes 70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable c8c568259772 xfs: don't include bnobt blocks when reserving free block pool cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks 919edbadebe1 xfs: drop async cache flushes from CIL commits.

Tests are being run through gce-xfstests with the 5.15 kernel config from xfstests-bld
(https://github.com/tytso/xfstests-bld/blob/master/kernel-configs/x86_
64-config-5.15). The configs being tested are the following:

xfs defaults
quota
quota 1k
v4
pmem and fsdax
realtime
8k directory blocks
external log
realtime and external log devices
realtime with 28k extents, external log devices
overlayfs atop xfs
overlayfs atop ext4
ext4 defaults

The test set will be run for each batch of backports, running each test 3 times, and if no new failures are seen compared to the same branch without the backports, the batch of patches will be deemed good. No regressions were seen for the first set of patches listed above when applied to 5.15.33. If new failures are seen during testing, a bisect can be run to find the offending commits, remove these from the batch, and confirm there are no remaining new failures. A bug report can be sent to indicate which commits would cause new test failures. The test results can be posted publicly after each run. The easiest option would be to send the test results to a mailing list, such as a google groups mailing list, similar to what syzkaller does, or directly to linux-xfs if it isn’t too spammy.

- Leah

#3 lore raw
Quoted 15 lines · Leah Rumancik

This thread had good timing :) I have been working on setting up some automated testing. Currently, 5.15.y is our priority so I have started working on this branch.

Patches are being selected by simply searching for the “Fixes” tag and applying if the commit-to-be-fixed is in the stable branch, but AUTOSEL would be nice, so I’ll start playing around with that. Amir, it would be nice to sync up the patch selection process. I can help share the load, especially for 5.15.

Selecting just the tagged “Fixes” for 5.15.y for patches through 5.17.2, 15 patches were found and applied - if there are no complaints about the testing setup, I can go ahead and send out this batch:

Quoted 15 lines

c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects 5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname 1090427bf18f xfs: remove xfs_inew_wait
089558bc7ba7 xfs: remove all COW fork extents when remounting readonly 7993f1a431bc xfs: only run COW extent recovery when there are no live extents 09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes 70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable c8c568259772 xfs: don't include bnobt blocks when reserving free block pool cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks 919edbadebe1 xfs: drop async cache flushes from CIL commits.

Please include:
9a5280b312e2 xfs: reorder iunlink remove operation in xfs_ifree

Thanks!

#4 lore raw
In reply to Leah Rumancik
Quoted 36 lines · Leah Rumancik
Quoted 10 lines · Darrick J. Wong

Hi everyone,

3. fstesting -- new patches proposed for stable branches shouldn't introduce new regressions, and ideally there would also be a regression test that would now pass. As Dave and I have stated in the past, fstests is a big umbrella of a test suite, which implies that A/B testing is the way to go. I think at least Zorro and I would like to improve the tagging in fstests to make it more obvious which tests contain enough randomness that they cannot be expected to behave 100% reliably.

It would be nice to find an agreement on testing requirements. I have attached some ideas on configs/number of tests/etc as well as the status of my work on 5.15 below.

Quoted 8 lines

a> I've been following the recent fstests threads, and it seems to me that there are really two classes of users -- sustaining people who want fstests to run reliably so they can tell if their backports have broken anything; and developers, who want the randomness to try to poke into dusty corners of the filesystem. Can we make it easier to associate random bits of data (reliability rates, etc.) with a given fstests configuration? And create a test group^Wtag for the tests that rely on RNGs to shake things up?

This would be great!

Quoted 5 lines

Thoughts? Flames?

--D

This thread had good timing :) I have been working on setting up some automated testing. Currently, 5.15.y is our priority so I have started working on this branch.

Quoted 6 lines

Patches are being selected by simply searching for the “Fixes” tag and applying if the commit-to-be-fixed is in the stable branch, but AUTOSEL would be nice, so I’ll start playing around with that. Amir, it would be nice to sync up the patch selection process. I can help share the load, especially for 5.15.

I would like that :)

Quoted 5 lines

Selecting just the tagged “Fixes” for 5.15.y for patches through 5.17.2, 15 patches were found and applied - if there are no complaints about the testing setup, I can go ahead and send out this batch:

Quoted 16 lines

c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects 5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname 1090427bf18f xfs: remove xfs_inew_wait
089558bc7ba7 xfs: remove all COW fork extents when remounting readonly 7993f1a431bc xfs: only run COW extent recovery when there are no live extents 09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes 70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable c8c568259772 xfs: don't include bnobt blocks when reserving free block pool cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks 919edbadebe1 xfs: drop async cache flushes from CIL commits.

Here are my selection for v5.15..v5.17:

* 1cd231d9fdb1 - (tag: xfs-5.10.y-7) xfs: use setattr_copy to set vfs inode attributes
* af09d052db41 - xfs: fallocate() should call file_modified()
* 0daebb90e096 - xfs: reject crazy array sizes being fed to XFS_IOC_GETBMAP*
* 35d876873c28 - xfs: prevent UAF in xfs_log_item_in_current_chkpt
* 796e9e00071d - xfs: xfs_log_force_lsn isn't passed a LSN
* fa33747dd25b - xfs: refactor xfs_file_fsync
* 374a05b9a2de - xfs: check sb_meta_uuid for dabuf buffer recovery
* 0b66f78d6af1 - (tag: xfs-5.10.y-6) xfs: remove all COW fork extents when remounting readonly
* 44caa4c7aaf4 - xfs: remove incorrect ASSERT in xfs_rename
* 4133fc82c95d - xfs: punch out data fork delalloc blocks on COW writeback failure

The branch of the moment is at:
https://github.com/amir73il/linux/commits/xfs-5.10.y
But I keep force pushing the branch and tags when dropping patches from the queue.

Note that only half of those commits have Fixes: tag.
As I explained, I got to them by removing all the non-fixes and non-relevant commits and then tried to evaluate the remaining commits individually. This was only made scalable because I was working at the patch series level and not at commit level, although at many times, a single fix patch was selected from within a non-fix series.

Note that many of the fixes that you selected are impact waves of big performance improvements that got merged after 5.10, so were not relevant for my 5.10.y selection.

Thanks,
Amir.

#5 lore raw
In reply to Leah Rumancik
Quoted 46 lines · Leah Rumancik
Quoted 10 lines · Darrick J. Wong

Hi everyone,

3. fstesting -- new patches proposed for stable branches shouldn't introduce new regressions, and ideally there would also be a regression test that would now pass. As Dave and I have stated in the past, fstests is a big umbrella of a test suite, which implies that A/B testing is the way to go. I think at least Zorro and I would like to improve the tagging in fstests to make it more obvious which tests contain enough randomness that they cannot be expected to behave 100% reliably.

It would be nice to find an agreement on testing requirements. I have attached some ideas on configs/number of tests/etc as well as the status of my work on 5.15 below.

Quoted 8 lines

a> I've been following the recent fstests threads, and it seems to me that there are really two classes of users -- sustaining people who want fstests to run reliably so they can tell if their backports have broken anything; and developers, who want the randomness to try to poke into dusty corners of the filesystem. Can we make it easier to associate random bits of data (reliability rates, etc.) with a given fstests configuration? And create a test group^Wtag for the tests that rely on RNGs to shake things up?

This would be great!

Quoted 5 lines

Thoughts? Flames?

--D

This thread had good timing :) I have been working on setting up some automated testing. Currently, 5.15.y is our priority so I have started working on this branch.

Patches are being selected by simply searching for the “Fixes” tag and applying if the commit-to-be-fixed is in the stable branch, but AUTOSEL would be nice, so I’ll start playing around with that. Amir, it would be nice to sync up the patch selection process. I can help share the load, especially for 5.15.

Selecting just the tagged “Fixes” for 5.15.y for patches through 5.17.2, 15 patches were found and applied - if there are no complaints about the testing setup, I can go ahead and send out this batch:

Quoted 16 lines

c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects 5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname 1090427bf18f xfs: remove xfs_inew_wait
089558bc7ba7 xfs: remove all COW fork extents when remounting readonly 7993f1a431bc xfs: only run COW extent recovery when there are no live extents 09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes 70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable c8c568259772 xfs: don't include bnobt blocks when reserving free block pool cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks 919edbadebe1 xfs: drop async cache flushes from CIL commits.

In our experience, we found that some of the patches which fix bugs would not have the associated "Fixes" tag. Hence I am currently using the script
https://gist.github.com/chandanr/c1e3affdb06eb2e025f955e7a77b2338 to identify such commits along with the commits which have the "Fixes" tag.

The following command line obtains the list of commits from v5.18,

# list-xfs-fix-commits.sh v5.17 v5.18
--- Actual fixes ---
1:  eba0549bc7d10
2:  e014f37db1a2d
3:  70447e0ad9781
4:  c8c5682597727
5:  cd6f79d1fb324
6:  919edbadebe17
7:  9a5280b312e2e

---- Possible fixes; Along with matching regex ----

1:  871b9316e7a77: bug
2:  41667260bc84d: bug
3:  83a44a4f47ad2: fix
4:  a9a4bc8c76d74: rac.+
5:  dbd0f5299302f: rac.+
6:  941fbdfd6dd0f: rac.+
7:  01728b44ef1b7: bug
8:  b9b1335e64030: fix
9:  82be38bcf8a2e: fix
10:  d2d7c0473586d: fix
11:  ab9c81ef321f9: assert
12:  b5f17bec1213a: rac.+
13:  41e6362183589: fix
14:  3c4cb76bce438: rac.+
15:  5652ef31705f2: fail

I go through each commit in the "Possible fixes" section and determine if any of those need to be backported.

Signature · 1 line
chandan
#6 lore raw
In reply to Darrick J. Wong
Quoted 4 lines · Darrick J. Wong

2. Some other tag for patches that could be a fix, but need a few months to soak. This is targetted at (b), since I'm terrible at remembering that there are patches that are reaching ripeness.

What I'd suggest here is a simple "Stable-Soak: <date>|<release>" tag. It wouldn't need to be official, and so we don't need to get the blessing of the Stable Tree maintainers; it would just be something that would be honored by the "XFS LTS backport cabal".

Quoted 8 lines

a> I've been following the recent fstests threads, and it seems to me that there are really two classes of users -- sustaining people who want fstests to run reliably so they can tell if their backports have broken anything; and developers, who want the randomness to try to poke into dusty corners of the filesystem. Can we make it easier to associate random bits of data (reliability rates, etc.) with a given fstests configuration? And create a test group^Wtag for the tests that rely on RNGs to shake things up?

In my experience, tests that have flaky results fall into two categories; ones that are trying to deal traditional fuzzing, and those that are running stress tests either by themselves, or as antagonists against some other operation --- e.g., running fstress while trying to do an online resize, or why trying to shut down the file system, etc.

Some of these stress tests do use a PRNG, but even if we fixed the seed to some value (such as 0), many of the test results would stlil be potentially flaky. These test results also tend to be very timing dependant; so these are the tests that whose failure rate varies depending on whether the test devices are on a loop device, eMMC flash device, HDD, SSD, or various cloud virtual block devices, such as AWS's EBS or or GCE's PD devices.

These tests are often very useful, since if we are missing a lock when accessing some data structure, these tests which use stress test programs are the most likely way noticing such problems. So I don't think we would want to exclude them; and if we're only excluding those tests which are doing fuzz testing, I'm not sure it's really move the needle.

Quoted 4 lines

b> Testing relies very heavily on being able to spin up a lot of testing resources. Can/should we make it easier for people with a kernel.org account to get free(ish) cloud accounts with the LF members who are also cloud vendors?

If anyone wants to use gce-xfstests, I'm happy to work on sponsoring some GCE credits for that purpose. One of the nice things about gce-xfstests is that Test VM's only are left running when actually running a test. Once a test is finished, the VM shuts itself down. And if we want to run a number of file system configs, we can spawn a dozen VM's, one for each fsconfig, and when they are done, each VM shuts itself down except for a small test test manager which collates the results into a single report. This makes gce-xfstests much more cost efficient that those schemes which keeps a VM up and running at all times, whether it is running tests or not.

Cheers,

- Ted