xfstests: what do we need to do to make it better?
Given that more people are using xfstests and developing tests, we need to consider how to make it friendlier to hack on. The current structure of the tree is difficult to work with, the way tests are organised and numbered make it difficult to co-ordinate new tests and results in patch conflicts, etc.
We also see problems arising from people not really understanding how the xfstests harness is designed and how it really is supposed to work, so an overview of the underlying principles of operation would probably be helpful to a lot of people. It will also save review and rework time if we can avoid having people make the same mistakes the first time they submit tests....
I'd also like to discuss some potential infrastructure changes to make it easier to add new tests without conflicts with others developing new tests. Some of the ideas Christoph and I have previously tossed around include:
- break tests up into groups in their own subdirectories. e.g. generic tests, xfs/ext4/btrfs specific tests, stress tests, performance tests, large FS tests, etc
- change the way we define groups of tests so we don't have a single registry of tests and their groups
- allow different naming of tests, such as desciptive text names rather than just plain numbers
- allow duplicate test names in different groups
I'm sure that other users of xfstests will have some ideas on how to improve it for the way they run it, so I'd like to gather and incorporate these ideas into any structural change we make to xfstests.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 5 lines · Dave Chinner
Given that more people are using xfstests and developing tests, we need to consider how to make it friendlier to hack on. The current structure of the tree is difficult to work with, the way tests are organised and numbered make it difficult to co-ordinate new tests and results in patch conflicts, etc.
Coordination of numbers is not a big deal, the test names/numbers can be easily fixed up at commit time. I also thought that the numbers--though meaningless on their own--also avoided having to decide where a particular test belongs. I.e., a test that exercises several categories of things (maybe preallocation, quota, and ENOSPC) won't be hidden in any sort of "enospc" test directory.
I do think the growing number of tests is making it a bit unwieldy though, so I think some sort of reorganization is a good plan.
Quoted 6 lines
We also see problems arising from people not really understanding how the xfstests harness is designed and how it really is supposed to work, so an overview of the underlying principles of operation would probably be helpful to a lot of people. It will also save review and rework time if we can avoid having people make the same mistakes the first time they submit tests....
This is very important. And the gist of it ought to be captured somewhere if it is not already.
Quoted 5 lines
I'd also like to discuss some potential infrastructure changes to make it easier to add new tests without conflicts with others developing new tests. Some of the ideas Christoph and I have previously tossed around include:
Quoted 8 lines
- break tests up into groups in their own subdirectories. e.g. generic tests, xfs/ext4/btrfs specific tests, stress tests, performance tests, large FS tests, etc
- change the way we define groups of tests so we don't have a single registry of tests and their groups
- allow different naming of tests, such as desciptive text names rather than just plain numbers
- allow duplicate test names in different groups
Despite what I said above, I don't disagree with any of this. Perhaps the tests can be buried in one or more subdirectories, but each FSTYP defines its own groups file to drive testing.
Quoted 4 lines
I'm sure that other users of xfstests will have some ideas on how to improve it for the way they run it, so I'd like to gather and incorporate these ideas into any structural change we make to xfstests.
Should be a good discussion. It might be useful to have a proposal or two to work with as a starting point, or maybe an outline of the types of changes (naming, directory structure, etc.), to help keep things focused.
-Alex
Quoted 3 lines
Cheers,
Dave.
Quoted 11 lines · Alex Elder
Quoted 5 lines · Dave Chinner
Given that more people are using xfstests and developing tests, we need to consider how to make it friendlier to hack on. The current structure of the tree is difficult to work with, the way tests are organised and numbered make it difficult to co-ordinate new tests and results in patch conflicts, etc.
Coordination of numbers is not a big deal, the test names/numbers can be easily fixed up at commit time. I also thought that the numbers--though meaningless on their own--also avoided having to decide where a particular test belongs.
Instead, we decide that via the global group file and via definitions within the test itself. Yes, the global group file allows tests to exist in multiple logical test groups, but that's something that canbe maintained even with a broken up deirectory heirarchy.
Quoted 3 lines
I.e., a test that exercises
several categories of things (maybe preallocation, quota, and ENOSPC) won't be hidden in any sort of "enospc" test directory.
No, but it will be a generic of filesystem specific test, so woul dbe located in such a subdirectory.
Basically, I'm thinking of something more fine grained like this:
generic/group /001 /005 .... xfs/group /002 /003 ....
So that we're not treading on each other's toes as much. Maintaining the group files still allows logical grouping of tests (like auto, quota, etc) so I don't see any change in functionality there. Maybe someone will have a good idea to improve the logical grouping implementation, but right now I'm not planning on killing that functionality at all...
Quoted 13 lines
Quoted 4 lines
make it easier to add new tests without conflicts with others developing new tests. Some of the ideas Christoph and I have previously tossed around include:
Quoted 8 lines
- break tests up into groups in their own subdirectories. e.g. generic tests, xfs/ext4/btrfs specific tests, stress tests, performance tests, large FS tests, etc
- change the way we define groups of tests so we don't have a single registry of tests and their groups
- allow different naming of tests, such as desciptive text names rather than just plain numbers
- allow duplicate test names in different groups
Quoted 3 lines
Despite what I said above, I don't disagree with any of this. Perhaps the tests can be buried in one or more subdirectories, but each FSTYP defines its own groups file to drive testing.
I think allowing duplicate tests is a bad thing - keeping the logical grouping effectively allows us to only keep one copy of a test no matter where it ends up.
Quoted 9 lines
Quoted 4 lines
I'm sure that other users of xfstests will have some ideas on how to improve it for the way they run it, so I'd like to gather and incorporate these ideas into any structural change we make to xfstests.
Should be a good discussion. It might be useful to have a proposal or two to work with as a starting point, or maybe an outline of the types of changes (naming, directory structure, etc.), to help keep things focused.
Sure, though I don't want to done so much that it's all set in stone before I found out what other people consider pain points that need to be fixed...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]