Add support for -q <n> unconditional loop

This series introduces "-q <n>" option to support unconditional looping. Along with this, it also brings some improvements to output data both, in stdout and in result.log.

(quoted diff) +0 -2

-q <n> option can be useful for fast unconditional looping (compared to
-I <n>) to test a certain flakey test and get it's pass/fail metrics.

This is similar to -L <n>, although with -L the test only loops if it fails the 1st time.
The individual patches can provide more details about other changes and improvements.

[v1] -> [v2]
1. This patch series removes the central config fs feature from [v1] (patch 4 patch 5) so that this can be reviewed separately. No functional changes.
2. Added R.B tag of Darrick is patch 1/5.

[v1] https://lore.kernel.org/all/[email protected]/

Nirjhar Roy (IBM) (3):
tests/selftest: Add a new pseudo flaky test.
check: Add -q <n> option to support unconditional looping.
check: Improve pass/fail metrics and section config output

  check                  | 113 ++++++++++++++++++++++++++++++-----------
  tests/selftest/007     |  21 ++++++++
  tests/selftest/007.out |   2 +
  3 files changed, 105 insertions(+), 31 deletions(-)

create mode 100755 tests/selftest/007
create mode 100644 tests/selftest/007.out

--
2.34.1

This patch adds -q <n> option through which one can run a given test <n> times unconditionally. It also prints pass/fail metrics at the end.

The advantage of this over -L <n> and -i/-I <n> is that:
a. -L <n> will not re-run a flakey test if the test passes for the first time. b. -I/-i <n> sets up devices during each iteration and hence slower.
Note -q <n> will override -L <n>.

Quoted 7 lines · Nirjhar Roy (IBM)

This patch adds -q <n> option through which one can run a given test <n> times unconditionally. It also prints pass/fail metrics at the end.

The advantage of this over -L <n> and -i/-I <n> is that:
a. -L <n> will not re-run a flakey test if the test passes for the first time. b. -I/-i <n> sets up devices during each iteration and hence slower.
Note -q <n> will override -L <n>.

I'm wondering if we need to keep the current behavior of -I/-i. The primary difference between them and how your proposed -q works is that instead of iterating over the section, your proposed option iterates over each test. So for example, if a section contains generic/001 and generic/002, iterating using -i 3 will do this:

generic/001
generic/002
generic/001
generic/002
generic/001
generic/002

While generic -q 3 would do this instead:

generic/001
generic/001
generic/001
generic/002
generic/002
generic/002

At least for all of the use cases that I can think of where I might use -i 3, -q 3 is strictly better. So instead of adding more options which change how we might do iterations, could we perhaps just replace -i with your new -q? And change -I so that it also works like -q, except if any test fails, that we stop?

- Ted

Quoted 7 lines · Nirjhar Roy (IBM)

This patch adds -q <n> option through which one can run a given test <n> times unconditionally. It also prints pass/fail metrics at the end.

The advantage of this over -L <n> and -i/-I <n> is that:
a. -L <n> will not re-run a flakey test if the test passes for the first time. b. -I/-i <n> sets up devices during each iteration and hence slower.
Note -q <n> will override -L <n>.

I'm wondering if we need to keep the current behavior of -I/-i. The primary difference between them and how your proposed -q works is that instead of iterating over the section, your proposed option iterates over each test. So for example, if a section contains generic/001 and generic/002, iterating using -i 3 will do this:

generic/001
generic/002
generic/001
generic/002
generic/001
generic/002

While generic -q 3 would do this instead:

generic/001
generic/001
generic/001
generic/002
generic/002
generic/002

At least for all of the use cases that I can think of where I might use -i 3, -q 3 is strictly better. So instead of adding more options which change how we might do iterations, could we perhaps just replace -i with your new -q? And change -I so that it also works like -q, except if any test fails, that we stop?

Quoted 8 lines · Theodore Ts'o

Quoted 7 lines · Nirjhar Roy (IBM)

This patch adds -q <n> option through which one can run a given test <n> times unconditionally. It also prints pass/fail metrics at the end.

The advantage of this over -L <n> and -i/-I <n> is that:
a. -L <n> will not re-run a flakey test if the test passes for the first time. b. -I/-i <n> sets up devices during each iteration and hence slower.
Note -q <n> will override -L <n>.

Quoted 5 lines

I'm wondering if we need to keep the current behavior of -I/-i. The primary difference between them and how your proposed -q works is that instead of iterating over the section, your proposed option iterates over each test. So for example, if a section contains generic/001 and generic/002, iterating using -i 3 will do this:

Yes, the motivation to introduce -q was to:

1. Make the re-run faster and not re-format the device. -i re-formats the device and hence is slightly slower.

2. To unconditionally loop a test - useful for scenarios when a flaky test doesn't fail for the first time (something that -L) does.

So, are saying that re-formatting a disk on every run, something that -i does, doesn't have much value and can be removed?

Quoted 7 lines

generic/001
generic/002
generic/001
generic/002
generic/001
generic/002

Quoted 8 lines

While generic -q 3 would do this instead:

generic/001
generic/001
generic/001
generic/002
generic/002
generic/002

Quoted 16 lines · Nirjhar Roy (IBM)

Quoted 8 lines · Theodore Ts'o

Quoted 7 lines · Nirjhar Roy (IBM)

This patch adds -q <n> option through which one can run a given test <n> times unconditionally. It also prints pass/fail metrics at the end.

The advantage of this over -L <n> and -i/-I <n> is that:
a. -L <n> will not re-run a flakey test if the test passes for the first time. b. -I/-i <n> sets up devices during each iteration and hence slower.
Note -q <n> will override -L <n>.

Quoted 5 lines

I'm wondering if we need to keep the current behavior of -I/-i. The primary difference between them and how your proposed -q works is that instead of iterating over the section, your proposed option iterates over each test. So for example, if a section contains generic/001 and generic/002, iterating using -i 3 will do this:

Yes, the motivation to introduce -q was to:

Quoted 2 lines

1. Make the re-run faster and not re-format the device. -i re-formats the device and hence is slightly slower.

Why does -i reformat the test device on every run in your setup? i.e. if the FSTYP is not changing from iteration to iteration, then each iteration should not reformat the test device at all. Unless, of course, you have told it to do so via the RECREATE_TEST_DEV env variable....

Hence it seems to me like this is working around some other setup or section iteration problem here...

Quoted 2 lines

2. To unconditionally loop a test - useful for scenarios when a flaky test doesn't fail for the first time (something that -L) does.

That's what -i does. it will unconditionally loop over the specified tests N times regardless of success or failure.

OTOH, -I will abort on first failure. i.e. to enable flakey tests to be run until it eventually fails and leave the corpse behind for debugging.

Quoted 2 lines

So, are saying that re-formatting a disk on every run, something that -i does, doesn't have much value and can be removed?

-i does not imply that the test device should be reformatted on every loop. If that is happening, that is likely a result of test config or environment conditions.

Can you tell us why the test device is getting reformatted on every iteration in your setup?

Quoted 8 lines

Quoted 8 lines

generic/001
generic/002
generic/001
generic/002
generic/001
generic/002

Quoted 8 lines

Quoted 8 lines

While generic -q 3 would do this instead:

generic/001
generic/001
generic/001
generic/002
generic/002
generic/002

There are arguments both for and against the different iteration orders. However, if there is no overriding reason to change the existing order of test execution, then we should not change the order or test execution....

-Dave.