Test cases to choose for demonstrating mm features or fixing mm bugs
Sending a patch to linux-mm today has become a complex task. One of the reasons for the complexity is a lack of fundamental expectation of what tests to run.
Mel Gorman has a set of tests [1], but there is no easy way to select what tests to run. Some of them are proprietary (spec*), but others have varying run times. A single line change may require hours or days of testing, add to that complexity of configuration. It requires a lot of tweaking and frequent test spawning to settle down on what to run, what configuration to choose and benefit to show.
The proposal is to have a discussion on how to design a good sanity test suite for the mm subsystem, which could potentially include OOM test cases and known problem patterns with proposed changes
It would be great if we could discuss that in the summit this time, all members are welcome and encouraged to participate.
References:
Quoted 11 lines · Balbir Singh
Sending a patch to linux-mm today has become a complex task. One of the reasons for the complexity is a lack of fundamental expectation of what tests to run.
Mel Gorman has a set of tests [1], but there is no easy way to select what tests to run. Some of them are proprietary (spec*), but others have varying run times. A single line change may require hours or days of testing, add to that complexity of configuration. It requires a lot of tweaking and frequent test spawning to settle down on what to run, what configuration to choose and benefit to show.
Quoted 3 lines
The proposal is to have a discussion on how to design a good sanity test suite for the mm subsystem, which could potentially include OOM test cases and known problem patterns with proposed changes
I am not sure I follow. So what is the problem you would like to solve. If tests are taking too long then there is a good reason for that most probably. Are you thinking of any specific tests which should be run or even included to MM tests or similar?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 12 lines · Michal Hocko
Quoted 11 lines · Balbir Singh
Sending a patch to linux-mm today has become a complex task. One of the reasons for the complexity is a lack of fundamental expectation of what tests to run.
Mel Gorman has a set of tests [1], but there is no easy way to select what tests to run. Some of them are proprietary (spec*), but others have varying run times. A single line change may require hours or days of testing, add to that complexity of configuration. It requires a lot of tweaking and frequent test spawning to settle down on what to run, what configuration to choose and benefit to show.
Quoted 8 lines
Quoted 3 lines
The proposal is to have a discussion on how to design a good sanity test suite for the mm subsystem, which could potentially include OOM test cases and known problem patterns with proposed changes
I am not sure I follow. So what is the problem you would like to solve. If tests are taking too long then there is a good reason for that most probably. Are you thinking of any specific tests which should be run or even included to MM tests or similar?
Let me elaborate, everytime I think I find something interesting, in terms of something to develop/fix, I think of how to test the changes. I think for well established code (such as reclaim) or even other features, it's hard to find good test cases to run as a base to ensure that
1. There is good coverage of tests against the changes
2. The right test cases have been run from a performance perspective
The reason I brought up the time was not the time for a single test, but all the tests cumulative in the absence of good guidance for
(1) and (2) above.
IOW, what guidance can we provide to patch writers and bug fixers in terms of what testing to carry out? How do we avoid biases in results and ensure consistency?
Balbir Singh.
Quoted 30 lines · Balbir Singh
Quoted 12 lines · Michal Hocko
Quoted 11 lines · Balbir Singh
Sending a patch to linux-mm today has become a complex task. One of the reasons for the complexity is a lack of fundamental expectation of what tests to run.
Mel Gorman has a set of tests [1], but there is no easy way to select what tests to run. Some of them are proprietary (spec*), but others have varying run times. A single line change may require hours or days of testing, add to that complexity of configuration. It requires a lot of tweaking and frequent test spawning to settle down on what to run, what configuration to choose and benefit to show.
Quoted 8 lines
Quoted 3 lines
The proposal is to have a discussion on how to design a good sanity test suite for the mm subsystem, which could potentially include OOM test cases and known problem patterns with proposed changes
I am not sure I follow. So what is the problem you would like to solve. If tests are taking too long then there is a good reason for that most probably. Are you thinking of any specific tests which should be run or even included to MM tests or similar?
Let me elaborate, everytime I think I find something interesting, in terms of something to develop/fix, I think of how to test the changes. I think for well established code (such as reclaim) or even other features, it's hard to find good test cases to run as a base to ensure that
1. There is good coverage of tests against the changes
2. The right test cases have been run from a performance perspective
Quoted 7 lines
The reason I brought up the time was not the time for a single test, but all the tests cumulative in the absence of good guidance for
(1) and (2) above.IOW, what guidance can we provide to patch writers and bug fixers in terms of what testing to carry out? How do we avoid biases in results and ensure consistency?
Well, I am afraid that there is no reference workload for the reclaim behavior or many other heuristics MM uses. This will always be workload dependant. Mel's mm-tests have a wider variety of workloads. There might be more of course. The most important part is how those represent real workload people do care about.
Abstracting workloads which are not in the test suits yet is definitely a step in the right direction.
Signature · 2 lines
Michal Hocko SUSE Labs