Skip to contentSkip to conversation

tst_test: Add min_runtime to control lower bound of scaled runtime

PATCH Superseded Li Wang via ltp 2 participants 8 messages Jun 23, 2025 – Jun 24, 2025 Library code

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 tst_test: Add min_runtime to control lower bound of scaled runtime

When LTP_RUNTIME_MUL is set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too aggressively. For some tests especially those relying on probabilistic triggers or sampling (e.g., fuzzy sync, CVE stress loops, long fault injection), this can result in too few iterations or missed conditions, leading to unreliable results.

This patch introduces a new field: .min_runtime;

If set, this specifies the minimum allowed runtime (in seconds) after applying the LTP_RUNTIME_MUL scaling factor. The final runtime is calculated as:

MAX(runtime * LTP_RUNTIME_MUL, min_runtime)

If min_runtime is not set, a default minimum of 1 second is enforced.

This approach replaces the need for special flags such as TST_NO_FRAC_RUNTIME_MUL and provides a more systematic, flexible solution.

The following tests are updated to set .min_runtime:
- preadv203
- cve-2016-7117
- tst_fuzzy_sync01
- tst_fuzzy_sync02

---

Notes:
v1 --> v2:
* replace the .flags by a new min_runtime filed
* set 1 second as the default minimal runtime regardless LTP_RUNTIME_MUL value

 include/tst_test.h                            |  9 +++++++++
 lib/newlib_tests/tst_fuzzy_sync01.c           |  2 +-
 lib/newlib_tests/tst_fuzzy_sync02.c           |  2 +-
 lib/tst_test.c                                | 11 +++++++++--
 testcases/cve/cve-2016-7117.c                 |  2 +-
 testcases/kernel/syscalls/preadv2/preadv203.c |  2 +-
 6 files changed, 22 insertions(+), 6 deletions(-)
include/tst_test.h +9 -0
index 6fd8cbae3..9c21c1728 100644
@@ -460,6 +460,14 @@ struct tst_fs {
  *           (e.g., TIMEOUT_MUL), ensuring consistent test duration across
  *           different environments (e.g., debug vs. stock kernels).
  *
+ * @min_runtime: Optional lower bound (in seconds) applied after runtime is
+ *           scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller
+ *           than this value, it will be clamped up to min_runtime.
+ *           This is useful for tests that require a minimum execution
+ *           time to gather sufficient samples or trigger events (e.g.,
+ *           probabilistic or fuzzy synchronization tests).
+ *           If not set, a default minimum of 1 second is enforced.
+ *
  * @setup: Setup callback is called once at the start of the test in order to
  *         prepare the test environment.
  *
@@ -584,6 +592,7 @@ struct tst_fs {
 
 	int timeout;
 	int runtime;
+	int min_runtime;
 
 	void (*setup)(void);
 	void (*cleanup)(void);
lib/newlib_tests/tst_fuzzy_sync01.c +1 -1
index b1390f460..16f30de57 100644
@@ -246,5 +246,5 @@ static struct tst_test test = {
 	.test = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 150,
+	.min_runtime = 150,
 };
lib/newlib_tests/tst_fuzzy_sync02.c +1 -1
index bc079f6ff..3b0fb8a9b 100644
@@ -223,5 +223,5 @@ static struct tst_test test = {
 	.test = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 150,
+	.min_runtime = 150,
 };
lib/tst_test.c +9 -2
index 495e022f7..75a78be9a 100644
@@ -627,10 +627,14 @@ static void parse_mul(float *mul, const char *env_name, float min, float max)
 static int multiply_runtime(unsigned int runtime)
 {
 	static float runtime_mul = -1;
+	int min_runtime = 1;
+
+	if (tst_test->min_runtime)
+		min_runtime = MAX(1, tst_test->min_runtime);
 
 	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
 
-	return runtime * runtime_mul;
+	return MAX(runtime * runtime_mul, min_runtime);
 }
 
 static struct option {
@@ -662,7 +666,7 @@ static void print_help(void)
 	fprintf(stderr, "LTP_SINGLE_FS_TYPE       Specifies filesystem instead all supported (for .all_filesystems)\n");
 	fprintf(stderr, "LTP_FORCE_SINGLE_FS_TYPE Testing only. The same as LTP_SINGLE_FS_TYPE but ignores test skiplist.\n");
 	fprintf(stderr, "LTP_TIMEOUT_MUL          Timeout multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >=1)\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >0)\n");
 	fprintf(stderr, "LTP_VIRT_OVERRIDE        Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
 	fprintf(stderr, "TMPDIR                   Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
 	fprintf(stderr, "\n");
@@ -1996,6 +2000,9 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 	uname(&uval);
 	tst_res(TINFO, "Tested kernel: %s %s %s", uval.release, uval.version, uval.machine);
 
+	if (tst_test->min_runtime)
+		tst_test->runtime =tst_test->min_runtime;
+
 	if (tst_test->runtime)
 		context->runtime = multiply_runtime(tst_test->runtime);
 
testcases/cve/cve-2016-7117.c +1 -1
index dbec2b28b..8c488f447 100644
@@ -151,7 +151,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 60,
+	.min_runtime = 60,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "34b88a68f26a"},
 		{"CVE", "2016-7117"},
testcases/kernel/syscalls/preadv2/preadv203.c +1 -1
index 72a35d3ab..472543e5c 100644
@@ -278,6 +278,6 @@ static struct tst_test test = {
 	.mntpoint = MNTPOINT,
 	.mount_device = 1,
 	.all_filesystems = 1,
-	.runtime = 60,
+	.min_runtime = 60,
 	.needs_root = 1,
 };
Signature · 5 lines
2.49.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

Hi!

Quoted 75 lines · Li Wang

When LTP_RUNTIME_MUL is set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too aggressively. For some tests especially those relying on probabilistic triggers or sampling (e.g., fuzzy sync, CVE stress loops, long fault injection), this can result in too few iterations or missed conditions, leading to unreliable results.

This patch introduces a new field: .min_runtime;

If set, this specifies the minimum allowed runtime (in seconds) after applying the LTP_RUNTIME_MUL scaling factor. The final runtime is calculated as:

MAX(runtime * LTP_RUNTIME_MUL, min_runtime)

If min_runtime is not set, a default minimum of 1 second is enforced.

This approach replaces the need for special flags such as TST_NO_FRAC_RUNTIME_MUL and provides a more systematic, flexible solution.

The following tests are updated to set .min_runtime:
- preadv203
- cve-2016-7117
- tst_fuzzy_sync01
- tst_fuzzy_sync02

---

Notes:
v1 --> v2:
* replace the .flags by a new min_runtime filed
* set 1 second as the default minimal runtime regardless LTP_RUNTIME_MUL value

  include/tst_test.h                            |  9 +++++++++
  lib/newlib_tests/tst_fuzzy_sync01.c           |  2 +-
  lib/newlib_tests/tst_fuzzy_sync02.c           |  2 +-
  lib/tst_test.c                                | 11 +++++++++--
  testcases/cve/cve-2016-7117.c                 |  2 +-
  testcases/kernel/syscalls/preadv2/preadv203.c |  2 +-
  6 files changed, 22 insertions(+), 6 deletions(-)
include/tst_test.h +9 -0
index 6fd8cbae3..9c21c1728 100644
@@ -460,6 +460,14 @@ struct tst_fs {
   *           (e.g., TIMEOUT_MUL), ensuring consistent test duration across
   *           different environments (e.g., debug vs. stock kernels).
   *
+ * @min_runtime: Optional lower bound (in seconds) applied after runtime is
+ *           scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller
+ *           than this value, it will be clamped up to min_runtime.
+ *           This is useful for tests that require a minimum execution
+ *           time to gather sufficient samples or trigger events (e.g.,
+ *           probabilistic or fuzzy synchronization tests).
+ *           If not set, a default minimum of 1 second is enforced.
+ *
   * @setup: Setup callback is called once at the start of the test in order to
   *         prepare the test environment.
   *
@@ -584,6 +592,7 @@ struct tst_fs {
 
  	int timeout;
  	int runtime;
+	int min_runtime;
 
  	void (*setup)(void);
  	void (*cleanup)(void);
lib/newlib_tests/tst_fuzzy_sync01.c +0 -0
index b1390f460..16f30de57 100644

lib/newlib_tests/tst_fuzzy_sync01.c static struct tst_test test = { · line 249

Quoted 5 lines
lib/newlib_tests/tst_fuzzy_sync01.c +1 -1
@@ -246,5 +246,5 @@ static struct tst_test test = {
  	.test = run,
  	.setup = setup,
  	.cleanup = cleanup,
-	.runtime = 150,
+	.min_runtime = 150,

Lib tests don't need .min_runtime unless you specifically want to test that LTP_RUNTIME_MUL gets handled correctly.

Quoted 42 lines
lib/newlib_tests/tst_fuzzy_sync01.c +0 -0
  };
lib/newlib_tests/tst_fuzzy_sync02.c +1 -1
index bc079f6ff..3b0fb8a9b 100644
@@ -223,5 +223,5 @@ static struct tst_test test = {
  	.test = run,
  	.setup = setup,
  	.cleanup = cleanup,
-	.runtime = 150,
+	.min_runtime = 150,
  };
lib/tst_test.c +6 -2
index 495e022f7..75a78be9a 100644
@@ -627,10 +627,14 @@ static void parse_mul(float *mul, const char *env_name, float min, float max)
  static int multiply_runtime(unsigned int runtime)
  {
  	static float runtime_mul = -1;
+	int min_runtime = 1;
+
+	if (tst_test->min_runtime)
+		min_runtime = MAX(1, tst_test->min_runtime);
 
  	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
 
-	return runtime * runtime_mul;
+	return MAX(runtime * runtime_mul, min_runtime);
  }
 
  static struct option {
@@ -662,7 +666,7 @@ static void print_help(void)
  	fprintf(stderr, "LTP_SINGLE_FS_TYPE       Specifies filesystem instead all supported (for .all_filesystems)\n");
  	fprintf(stderr, "LTP_FORCE_SINGLE_FS_TYPE Testing only. The same as LTP_SINGLE_FS_TYPE but ignores test skiplist.\n");
  	fprintf(stderr, "LTP_TIMEOUT_MUL          Timeout multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >=1)\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >0)\n");
  	fprintf(stderr, "LTP_VIRT_OVERRIDE        Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
  	fprintf(stderr, "TMPDIR                   Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
  	fprintf(stderr, "\n");

lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct tst_test *self) · line 2004

Quoted 5 lines
lib/tst_test.c +2 -0
@@ -1996,6 +2000,9 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
  	uname(&uval);
  	tst_res(TINFO, "Tested kernel: %s %s %s", uval.release, uval.version, uval.machine);
 
+	if (tst_test->min_runtime)
+		tst_test->runtime =tst_test->min_runtime;

If .runtime is set, .min_runtime should not override it. Setting .runtime < .min_runtime should trigger TBROK during lib setup.

Quoted 9 lines
lib/tst_test.c +1 -0
+
  	if (tst_test->runtime)
  		context->runtime = multiply_runtime(tst_test->runtime);
 
testcases/cve/cve-2016-7117.c +0 -0
index dbec2b28b..8c488f447 100644

testcases/cve/cve-2016-7117.c static struct tst_test test = { · line 154

Quoted 5 lines
testcases/cve/cve-2016-7117.c +1 -1
@@ -151,7 +151,7 @@ static struct tst_test test = {
  	.test_all = run,
  	.setup = setup,
  	.cleanup = cleanup,
-	.runtime = 60,
+	.min_runtime = 60,

The .runtime value in fuzzy_sync tests is calibrated to reliably reproduce the bug. You can usually set much smaller .min_runtime based on how long it takes to run 1024 iterations of the fuzzy loop, with some safety margin (x2?).

Quoted 15 lines
testcases/cve/cve-2016-7117.c +0 -0
  	.tags = (const struct tst_tag[]) {
  		{"linux-git", "34b88a68f26a"},
  		{"CVE", "2016-7117"},
testcases/kernel/syscalls/preadv2/preadv203.c +1 -1
index 72a35d3ab..472543e5c 100644
@@ -278,6 +278,6 @@ static struct tst_test test = {
  	.mntpoint = MNTPOINT,
  	.mount_device = 1,
  	.all_filesystems = 1,
-	.runtime = 60,
+	.min_runtime = 60,
  	.needs_root = 1,
  };
Signature · 10 lines
Martin Doucha   [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw
Quoted 83 lines · Martin Doucha

Hi!

Quoted 2 lines · Li Wang

When LTP_RUNTIME_MUL is set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too

aggressively.

Quoted 1 line

For some tests especially those relying on probabilistic triggers or

sampling

Quoted 1 line

(e.g., fuzzy sync, CVE stress loops, long fault injection), this can

result in

Quoted 5 lines

too few iterations or missed conditions, leading to unreliable results.

This patch introduces a new field: .min_runtime;

If set, this specifies the minimum allowed runtime (in seconds) after

applying

Quoted 8 lines

the LTP_RUNTIME_MUL scaling factor. The final runtime is calculated as:

MAX(runtime * LTP_RUNTIME_MUL, min_runtime)

If min_runtime is not set, a default minimum of 1 second is enforced.

This approach replaces the need for special flags such as TST_NO_FRAC_RUNTIME_MUL and provides a more systematic, flexible

solution.

Quoted 31 lines

The following tests are updated to set .min_runtime:
- preadv203
- cve-2016-7117
- tst_fuzzy_sync01
- tst_fuzzy_sync02

---

Notes:
v1 --> v2:
* replace the .flags by a new min_runtime filed
* set 1 second as the default minimal runtime regardless LTP_RUNTIME_MUL value

  include/tst_test.h                            |  9 +++++++++
  lib/newlib_tests/tst_fuzzy_sync01.c           |  2 +-
  lib/newlib_tests/tst_fuzzy_sync02.c           |  2 +-
  lib/tst_test.c                                | 11 +++++++++--
  testcases/cve/cve-2016-7117.c                 |  2 +-
  testcases/kernel/syscalls/preadv2/preadv203.c |  2 +-
  6 files changed, 22 insertions(+), 6 deletions(-)
include/tst_test.h +0 -0
index 6fd8cbae3..9c21c1728 100644

include/tst_test.h struct tst_fs { · line 460

Quoted 1 line
include/tst_test.h +0 -0
@@ -460,6 +460,14 @@ struct tst_fs {
   *           (e.g., TIMEOUT_MUL), ensuring consistent test duration

across

include/tst_test.h struct tst_fs {

Quoted 3 lines
include/tst_test.h +1 -0
   *           different environments (e.g., debug vs. stock kernels).
   *
+ * @min_runtime: Optional lower bound (in seconds) applied after

runtime is

include/tst_test.h struct tst_fs {

Quoted 8 lines
include/tst_test.h +7 -0
+ *           scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller
+ *           than this value, it will be clamped up to min_runtime.
+ *           This is useful for tests that require a minimum execution
+ *           time to gather sufficient samples or trigger events (e.g.,
+ *           probabilistic or fuzzy synchronization tests).
+ *           If not set, a default minimum of 1 second is enforced.
+ *
   * @setup: Setup callback is called once at the start of the test in

order to

Review

Quoted 11 lines
include/tst_test.h +1 -0
   *         prepare the test environment.
   *
@@ -584,6 +592,7 @@ struct tst_fs {
 
      int timeout;
      int runtime;
+     int min_runtime;
 
      void (*setup)(void);
      void (*cleanup)(void);
(quoted diff) +0 -0

b/lib/newlib_tests/tst_fuzzy_sync01.c

Quoted 13 lines

lib/newlib_tests/tst_fuzzy_sync01.c static struct tst_test test = { · line 249

Quoted 9 lines

index b1390f460..16f30de57 100644

lib/newlib_tests/tst_fuzzy_sync01.c +1 -1
@@ -246,5 +246,5 @@ static struct tst_test test = {
      .test = run,
      .setup = setup,
      .cleanup = cleanup,
-     .runtime = 150,
+     .min_runtime = 150,

Lib tests don't need .min_runtime unless you specifically want to test that LTP_RUNTIME_MUL gets handled correctly.

I set .min_runtime for all fuzzy_sync tests that contain "pair.min_samples = XXX",
because the dedicated sampling times need more time and it is hard to tell the real minimal time, so I just reuse the old .runtime value.

And yes we can ignore this in lib tests, but I want to make them as examples.

Quoted 55 lines

Review

Quoted 3 lines
lib/newlib_tests/tst_fuzzy_sync01.c +0 -0
  };
(quoted diff) +0 -0

b/lib/newlib_tests/tst_fuzzy_sync02.c

lib/tst_test.c static void parse_mul(float *mul, const char

Quoted 15 lines

index bc079f6ff..3b0fb8a9b 100644

lib/newlib_tests/tst_fuzzy_sync02.c +1 -1
@@ -223,5 +223,5 @@ static struct tst_test test = {
      .test = run,
      .setup = setup,
      .cleanup = cleanup,
-     .runtime = 150,
+     .min_runtime = 150,
  };
lib/tst_test.c +0 -0
index 495e022f7..75a78be9a 100644
@@ -627,10 +627,14 @@ static void parse_mul(float *mul, const char

*env_name, float min, float max)

Quoted 16 lines
lib/tst_test.c +5 -1
  static int multiply_runtime(unsigned int runtime)
  {
      static float runtime_mul = -1;
+     int min_runtime = 1;
+
+     if (tst_test->min_runtime)
+             min_runtime = MAX(1, tst_test->min_runtime);
 
      parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
 
-     return runtime * runtime_mul;
+     return MAX(runtime * runtime_mul, min_runtime);
  }
 
  static struct option {

lib/tst_test.c static void print_help(void) · line 666

Quoted 1 line
lib/tst_test.c +0 -0
@@ -662,7 +666,7 @@ static void print_help(void)
      fprintf(stderr, "LTP_SINGLE_FS_TYPE       Specifies filesystem

instead all supported (for .all_filesystems)\n");

lib/tst_test.c static void print_help(void) · line 667

Quoted 1 line
lib/tst_test.c +0 -0
      fprintf(stderr, "LTP_FORCE_SINGLE_FS_TYPE Testing only. The same

as LTP_SINGLE_FS_TYPE but ignores test skiplist.\n");

lib/tst_test.c static void print_help(void) · line 668

Quoted 1 line
lib/tst_test.c +0 -0
      fprintf(stderr, "LTP_TIMEOUT_MUL          Timeout multiplier (must

be a number >=1)\n");

lib/tst_test.c static void print_help(void) · line 665

Quoted 1 line
lib/tst_test.c +0 -1
-     fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must

be a number >=1)\n");

lib/tst_test.c static void print_help(void) · line 669

Quoted 1 line
lib/tst_test.c +1 -0
+     fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must

be a number >0)\n");

lib/tst_test.c static void print_help(void) · line 670

Quoted 1 line
lib/tst_test.c +0 -0
      fprintf(stderr, "LTP_VIRT_OVERRIDE        Overrides virtual

machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");

lib/tst_test.c static void print_help(void) · line 671

Quoted 1 line
lib/tst_test.c +0 -0
      fprintf(stderr, "TMPDIR                   Base directory for

template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);

lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct

Quoted 2 lines
lib/tst_test.c +0 -0
      fprintf(stderr, "\n");
@@ -1996,6 +2000,9 @@ void tst_run_tcases(int argc, char *argv[], struct

tst_test *self)

lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct · line 2001

Quoted 2 lines
lib/tst_test.c +0 -0
      uname(&uval);
      tst_res(TINFO, "Tested kernel: %s %s %s", uval.release,
Quoted 9 lines

uval.version, uval.machine);

lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct tst_test *self) · line 2004

Quoted 3 lines
lib/tst_test.c +2 -0
+     if (tst_test->min_runtime)
+             tst_test->runtime =tst_test->min_runtime;

If .runtime is set, .min_runtime should not override it. Setting .runtime < .min_runtime should trigger TBROK during lib setup.

Not necessary, in most cases, we don't need to set both simultaneously. Even if .runtime is less than .min_runtime, MAX() will automatically choose .min_runtime. (but I strongly do not recommend set both)

Or, maybe here we can override if .runtime not being set to avoid lib complains no .runtime set failure.

- if (tst_test->min_runtime)
+ if (tst_test->min_runtime && !tst_test->runtime)

Quoted 7 lines

Review

Quoted 6 lines
lib/tst_test.c +1 -0
+
      if (tst_test->runtime)
              context->runtime = multiply_runtime(tst_test->runtime);
 
(quoted diff) +0 -0

b/testcases/cve/cve-2016-7117.c

Quoted 15 lines

testcases/cve/cve-2016-7117.c static struct tst_test test = { · line 154

Quoted 9 lines

index dbec2b28b..8c488f447 100644

testcases/cve/cve-2016-7117.c +1 -1
@@ -151,7 +151,7 @@ static struct tst_test test = {
      .test_all = run,
      .setup = setup,
      .cleanup = cleanup,
-     .runtime = 60,
+     .min_runtime = 60,

The .runtime value in fuzzy_sync tests is calibrated to reliably reproduce the bug. You can usually set much smaller .min_runtime based on how long it takes to run 1024 iterations of the fuzzy loop, with some safety margin (x2?).

In theory, yes, but the difficulty is thatthe calibration time depends on the various test platforms and system load conditions.

Here I only reset the .runtime to .min_runtime is for tests contain 'pair.min_smaples', which typically has a large value (e.g. 10000).

For those with the default 1024 iterations fuzzy_sync tests I did nothing for them, we may need more experiments to get a proper value.

Quoted 29 lines

Review

Quoted 5 lines
testcases/cve/cve-2016-7117.c +0 -0
      .tags = (const struct tst_tag[]) {
              {"linux-git", "34b88a68f26a"},
              {"CVE", "2016-7117"},
(quoted diff) +0 -0

b/testcases/kernel/syscalls/preadv2/preadv203.c

testcases/kernel/syscalls/preadv2/preadv203.c static struct tst_test test = { · line 283

Quoted 11 lines

index 72a35d3ab..472543e5c 100644

testcases/kernel/syscalls/preadv2/preadv203.c +1 -1
@@ -278,6 +278,6 @@ static struct tst_test test = {
      .mntpoint = MNTPOINT,
      .mount_device = 1,
      .all_filesystems = 1,
-     .runtime = 60,
+     .min_runtime = 60,
      .needs_root = 1,
  };

--
Martin Doucha [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

Signature · 5 lines
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw
In reply to Li Wang via ltp
Quoted 48 lines · Li Wang via ltp

When LTP_RUNTIME_MUL is set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too aggressively. For some tests especially those relying on probabilistic triggers or sampling (e.g., fuzzy sync, CVE stress loops, long fault injection), this can result in too few iterations or missed conditions, leading to unreliable results.

This patch introduces a new field: .min_runtime;

If set, this specifies the minimum allowed runtime (in seconds) after applying the LTP_RUNTIME_MUL scaling factor. The final runtime is calculated as:

MAX(runtime * LTP_RUNTIME_MUL, min_runtime)

If min_runtime is not set, a default minimum of 1 second is enforced.

This approach replaces the need for special flags such as TST_NO_FRAC_RUNTIME_MUL and provides a more systematic, flexible solution.

The following tests are updated to set .min_runtime:
- preadv203
- cve-2016-7117
- tst_fuzzy_sync01
- tst_fuzzy_sync02

---

Notes:
v1 --> v2:
* replace the .flags by a new min_runtime filed
* set 1 second as the default minimal runtime regardless LTP_RUNTIME_MUL value

 include/tst_test.h                            |  9 +++++++++
 lib/newlib_tests/tst_fuzzy_sync01.c           |  2 +-
 lib/newlib_tests/tst_fuzzy_sync02.c           |  2 +-
 lib/tst_test.c                                | 11 +++++++++--
 testcases/cve/cve-2016-7117.c                 |  2 +-
 testcases/kernel/syscalls/preadv2/preadv203.c |  2 +-
 6 files changed, 22 insertions(+), 6 deletions(-)
include/tst_test.h +0 -0
index 6fd8cbae3..9c21c1728 100644

include/tst_test.h struct tst_fs { · line 469

Quoted 10 lines
include/tst_test.h +7 -0
@@ -460,6 +460,14 @@ struct tst_fs {
  *           (e.g., TIMEOUT_MUL), ensuring consistent test duration across
  *           different environments (e.g., debug vs. stock kernels).
  *
+ * @min_runtime: Optional lower bound (in seconds) applied after runtime is
+ *           scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller
+ *           than this value, it will be clamped up to min_runtime.
+ *           This is useful for tests that require a minimum execution
+ *           time to gather sufficient samples or trigger events (e.g.,
+ *           probabilistic or fuzzy synchronization tests).
+ *           If not set, a default minimum of 1 second is enforced.

Base your logic once .min_runtime is set then .runtime value effectively becomes irrelevant. So i guess we need mention it in above description?

Quoted 104 lines
include/tst_test.h +2 -0
+ *
  * @setup: Setup callback is called once at the start of the test in order to
  *         prepare the test environment.
  *
@@ -584,6 +592,7 @@ struct tst_fs {
 
 	int timeout;
 	int runtime;
+	int min_runtime;
 
 	void (*setup)(void);
 	void (*cleanup)(void);
lib/newlib_tests/tst_fuzzy_sync01.c +1 -1
index b1390f460..16f30de57 100644
@@ -246,5 +246,5 @@ static struct tst_test test = {
 	.test = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 150,
+	.min_runtime = 150,
 };
lib/newlib_tests/tst_fuzzy_sync02.c +1 -1
index bc079f6ff..3b0fb8a9b 100644
@@ -223,5 +223,5 @@ static struct tst_test test = {
 	.test = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 150,
+	.min_runtime = 150,
 };
lib/tst_test.c +9 -2
index 495e022f7..75a78be9a 100644
@@ -627,10 +627,14 @@ static void parse_mul(float *mul, const char *env_name, float min, float max)
 static int multiply_runtime(unsigned int runtime)
 {
 	static float runtime_mul = -1;
+	int min_runtime = 1;
+
+	if (tst_test->min_runtime)
+		min_runtime = MAX(1, tst_test->min_runtime);
 
 	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
 
-	return runtime * runtime_mul;
+	return MAX(runtime * runtime_mul, min_runtime);
 }
 
 static struct option {
@@ -662,7 +666,7 @@ static void print_help(void)
 	fprintf(stderr, "LTP_SINGLE_FS_TYPE       Specifies filesystem instead all supported (for .all_filesystems)\n");
 	fprintf(stderr, "LTP_FORCE_SINGLE_FS_TYPE Testing only. The same as LTP_SINGLE_FS_TYPE but ignores test skiplist.\n");
 	fprintf(stderr, "LTP_TIMEOUT_MUL          Timeout multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >=1)\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >0)\n");
 	fprintf(stderr, "LTP_VIRT_OVERRIDE        Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
 	fprintf(stderr, "TMPDIR                   Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
 	fprintf(stderr, "\n");
@@ -1996,6 +2000,9 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 	uname(&uval);
 	tst_res(TINFO, "Tested kernel: %s %s %s", uval.release, uval.version, uval.machine);
 
+	if (tst_test->min_runtime)
+		tst_test->runtime =tst_test->min_runtime;
+
 	if (tst_test->runtime)
 		context->runtime = multiply_runtime(tst_test->runtime);
 
testcases/cve/cve-2016-7117.c +1 -1
index dbec2b28b..8c488f447 100644
@@ -151,7 +151,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 60,
+	.min_runtime = 60,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "34b88a68f26a"},
 		{"CVE", "2016-7117"},
testcases/kernel/syscalls/preadv2/preadv203.c +1 -1
index 72a35d3ab..472543e5c 100644
@@ -278,6 +278,6 @@ static struct tst_test test = {
 	.mntpoint = MNTPOINT,
 	.mount_device = 1,
 	.all_filesystems = 1,
-	.runtime = 60,
+	.min_runtime = 60,
 	.needs_root = 1,
 };
Signature · 5 lines
2.49.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
Quoted 8 lines · Wei Gao
Quoted 1 line

+ * @min_runtime: Optional lower bound (in seconds) applied after

runtime is

Quoted 6 lines
+ *           scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller
+ *           than this value, it will be clamped up to min_runtime.
+ *           This is useful for tests that require a minimum execution
+ *           time to gather sufficient samples or trigger events (e.g.,
+ *           probabilistic or fuzzy synchronization tests).
+ *           If not set, a default minimum of 1 second is enforced.
Quoted 4 lines

Base your logic once .min_runtime is set then .runtime value effectively becomes irrelevant.
So i guess we need mention it in above description?

Yes, combined with Martin's comment, we should at least ensure that .runtime is not overwritten when it exists.

+ if (tst_test->min_runtime && !tst_test->runtime)
+ tst_test->runtime = tst_test->min_runtime;

Also, I make the .min_runtime uses the value of .runtime (then remove .runtime) for those fuzzy_sync tests, that have a large 'pair.min_smaples' value.

But the default 1024 min_samples I still have no idea how long the .min_runtime needs. Maybe we can estimate and set .min_runtime on a slow system manually.

For others, which do not have strict requirements on execution time, we only need to set .runtime.

Signature · 5 lines
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw
Quoted 19 lines · Li Wang
Quoted 8 lines · Wei Gao
Quoted 1 line

+ * @min_runtime: Optional lower bound (in seconds) applied after

runtime is

Quoted 6 lines
+ *           scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller
+ *           than this value, it will be clamped up to min_runtime.
+ *           This is useful for tests that require a minimum execution
+ *           time to gather sufficient samples or trigger events (e.g.,
+ *           probabilistic or fuzzy synchronization tests).
+ *           If not set, a default minimum of 1 second is enforced.
Quoted 4 lines

Base your logic once .min_runtime is set then .runtime value effectively becomes irrelevant.
So i guess we need mention it in above description?

Yes, combined with Martin's comment, we should at least ensure that .runtime is not overwritten when it exists.

Review

Quoted 2 lines
(quoted diff) +2 -0
+       if (tst_test->min_runtime && !tst_test->runtime)
+               tst_test->runtime = tst_test->min_runtime;

Thanks for clarification. I think above logic is better.

Quoted 6 lines

Also, I make the .min_runtime uses the value of .runtime (then remove .runtime) for those fuzzy_sync tests, that have a large 'pair.min_smaples' value.

Quoted 6 lines

But the default 1024 min_samples I still have no idea how long the .min_runtime needs. Maybe we can estimate and set .min_runtime on a slow system manually.

For others, which do not have strict requirements on execution time, we only need to set .runtime.

No experience on *good* value for .min_runtime. But i guess reusing value of .runtime is the safest option.

Quoted 4 lines
Signature · 3 lines
-- 
Regards,
Li Wang
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
In reply to Li Wang via ltp

Hi All,

Quoted 4 lines · Li Wang

But the default 1024 min_samples I still have no idea how long the .min_runtime needs. Maybe we can estimate and set .min_runtime on a slow system manually.

I choose to run the fuzzy_sync tests on a Cortex-A55 CPU with only one core assigned, using the taskset command to simulate an extreme single-core execution scenario.

Tested on:

    Cortex-A55, 1.2GHz
    Linux 5.14, aarch64

For example:
time taskset -c 0 ./cve-2016-7117 (hacked the test only do sampling)

This setup allows me to evaluate how long the sampling phase takes under constrained conditions.

Based on the observed sampling duration, I apply the following policy for setting .min_runtime:

1. If the sampling time is very short (less than 1 second), I simply set .min_runtime = 2.

2. If the sampling time is longer than 2 seconds but still less than .runtime, I set .min_runtime to twice the sampling time, while keeping .runtime unchanged.

3. If the sampling time exceeds .runtime, I also double the sampling time for .min_runtime, but leave .runtime as is.

Signature · 5 lines
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw
Quoted 30 lines · Li Wang

Hi All,

Quoted 4 lines · Li Wang

But the default 1024 min_samples I still have no idea how long the .min_runtime needs. Maybe we can estimate and set .min_runtime on a slow system manually.

I choose to run the fuzzy_sync tests on a Cortex-A55 CPU with only one core assigned, using the taskset command to simulate an extreme single-core execution scenario.

Tested on:

    Cortex-A55, 1.2GHz
    Linux 5.14, aarch64

For example:
time taskset -c 0 ./cve-2016-7117 (hacked the test only do sampling)

This setup allows me to evaluate how long the sampling phase takes under constrained conditions.

Based on the observed sampling duration, I apply the following policy for setting .min_runtime:

1. If the sampling time is very short (less than 1 second), I simply set .min_runtime = 2.

Quoted 7 lines

2. If the sampling time is longer than 2 seconds but still less than .runtime, I set .min_runtime to twice the sampling time, while keeping .runtime unchanged.

3. If the sampling time exceeds .runtime, I also double the sampling time for .min_runtime, but leave .runtime as is.

3. If the sampling time exceeds .runtime, I simply set .min_runtime to the .runtime value and remove .runtime.

Signature · 5 lines
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp