Skip to contentSkip to conversation

tst_test: Add min_runtime to control lower bound of scaled runtime

PATCH In review Li Wang via ltp 2 participants 5 messages Jun 24, 2025 – Jul 17, 2025 Library code

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 tst_test: Add min_runtime to control lower bound of scaled runtime

When LTP_RUNTIME_MUL is set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too aggressively. For some tests especially those relying on probabilistic triggers or sampling (e.g., fuzzy sync, CVE stress loops, long fault injection), this can result in too few iterations or missed conditions, leading to unreliable results.

This patch introduces a new field: .min_runtime;

If set, this specifies the minimum allowed runtime (in seconds) after applying the LTP_RUNTIME_MUL scaling factor. The final runtime is calculated as:

MAX(runtime * LTP_RUNTIME_MUL, min_runtime)

If min_runtime is not set, a default minimum of 1 second is enforced.

This approach replaces the need for special flags such as TST_NO_FRAC_RUNTIME_MUL and provides a more systematic, flexible solution.

To validate this logic, I ran the fuzzy_sync tests on a Cortex-A55 CPU at 1.2GHz, running Linux 5.14 (aarch64), with only a single core assigned using taskset to simulate a worst-case, single-core execution scenario:

    time taskset -c 0 ./cve-2016-7117

(The test was locally modified to only perform the sampling phase.)

This setup allows accurate measurement of sampling duration under heavily constrained CPU conditions.

Based on the observed sampling time, I apply the following policy for setting .min_runtime in affected tests:

1. If the sampling time is very short (< 1s), set .min_runtime = 2.

2. If the sampling time is > 2s but still less than .runtime, set .min_runtime = 2 * sampling_time, and leave .runtime unchanged.

3. If the sampling time exceeds .runtime, set .min_runtime = .runtime and remove .runtime.

This ensures that the sampling phase is not prematurely interrupted due to aggressive runtime scaling, which is especially important in CI and low-power platforms.

This change improves test reliability under constrained runtime conditions without sacrificing flexibility for fast-path test execution.

Tested on:
- Cortex-A55, 1.2GHz
- Linux 5.14.0-587.536.el9iv.aarch64

---
 include/tst_test.h                                    |  9 +++++++++
 lib/tst_test.c                                        | 11 +++++++++--
 testcases/cve/cve-2014-0196.c                         |  2 +-
 testcases/cve/cve-2016-7117.c                         |  1 +
 testcases/cve/cve-2017-2671.c                         |  1 +
 testcases/kernel/crypto/af_alg07.c                    |  1 +
 testcases/kernel/pty/pty03.c                          |  1 +
 testcases/kernel/pty/pty05.c                          |  1 +
 testcases/kernel/pty/pty07.c                          |  1 +
 testcases/kernel/sound/snd_seq01.c                    |  1 +
 testcases/kernel/sound/snd_timer01.c                  |  1 +
 testcases/kernel/syscalls/bind/bind06.c               |  2 +-
 testcases/kernel/syscalls/inotify/inotify09.c         |  1 +
 testcases/kernel/syscalls/ipc/shmctl/shmctl05.c       |  2 +-
 testcases/kernel/syscalls/preadv2/preadv203.c         |  2 +-
 testcases/kernel/syscalls/sendmsg/sendmsg03.c         |  1 +
 testcases/kernel/syscalls/setsockopt/setsockopt06.c   |  2 +-
 testcases/kernel/syscalls/setsockopt/setsockopt07.c   |  2 +-
 testcases/kernel/syscalls/timerfd/timerfd_settime02.c |  1 +
 testcases/kernel/syscalls/writev/writev03.c           |  1 +
 testcases/network/can/cve/can_bcm01.c                 |  2 +-
 testcases/network/packet/fanout01.c                   |  2 +-
 testcases/network/sockets/vsock01.c                   |  1 +
 23 files changed, 39 insertions(+), 10 deletions(-)
include/tst_test.h +9 -0
index 6fd8cbae3..9c21c1728 100644
@@ -460,6 +460,14 @@ struct tst_fs {
  *           (e.g., TIMEOUT_MUL), ensuring consistent test duration across
  *           different environments (e.g., debug vs. stock kernels).
  *
+ * @min_runtime: Optional lower bound (in seconds) applied after runtime is
+ *           scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller
+ *           than this value, it will be clamped up to min_runtime.
+ *           This is useful for tests that require a minimum execution
+ *           time to gather sufficient samples or trigger events (e.g.,
+ *           probabilistic or fuzzy synchronization tests).
+ *           If not set, a default minimum of 1 second is enforced.
+ *
  * @setup: Setup callback is called once at the start of the test in order to
  *         prepare the test environment.
  *
@@ -584,6 +592,7 @@ struct tst_fs {
 
 	int timeout;
 	int runtime;
+	int min_runtime;
 
 	void (*setup)(void);
 	void (*cleanup)(void);
lib/tst_test.c +9 -2
index 495e022f7..bbe39b549 100644
@@ -627,10 +627,14 @@ static void parse_mul(float *mul, const char *env_name, float min, float max)
 static int multiply_runtime(unsigned int runtime)
 {
 	static float runtime_mul = -1;
+	int min_runtime = 1;
+
+	if (tst_test->min_runtime)
+		min_runtime = MAX(1, tst_test->min_runtime);
 
 	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
 
-	return runtime * runtime_mul;
+	return MAX(runtime * runtime_mul, min_runtime);
 }
 
 static struct option {
@@ -662,7 +666,7 @@ static void print_help(void)
 	fprintf(stderr, "LTP_SINGLE_FS_TYPE       Specifies filesystem instead all supported (for .all_filesystems)\n");
 	fprintf(stderr, "LTP_FORCE_SINGLE_FS_TYPE Testing only. The same as LTP_SINGLE_FS_TYPE but ignores test skiplist.\n");
 	fprintf(stderr, "LTP_TIMEOUT_MUL          Timeout multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >=1)\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL          Runtime multiplier (must be a number >0)\n");
 	fprintf(stderr, "LTP_VIRT_OVERRIDE        Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
 	fprintf(stderr, "TMPDIR                   Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
 	fprintf(stderr, "\n");
@@ -1996,6 +2000,9 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 	uname(&uval);
 	tst_res(TINFO, "Tested kernel: %s %s %s", uval.release, uval.version, uval.machine);
 
+	if (tst_test->min_runtime && !tst_test->runtime)
+		tst_test->runtime = tst_test->min_runtime;
+
 	if (tst_test->runtime)
 		context->runtime = multiply_runtime(tst_test->runtime);
 
testcases/cve/cve-2014-0196.c +1 -1
index 5e5b55e07..d9c4fb7eb 100644
@@ -144,7 +144,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = run,
-	.runtime = 60,
+	.min_runtime = 60,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "4291086b1f08"},
 		{"CVE", "2014-0196"},
testcases/cve/cve-2016-7117.c +1 -0
index dbec2b28b..d9c44a0b3 100644
@@ -152,6 +152,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.runtime = 60,
+	.min_runtime = 12,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "34b88a68f26a"},
 		{"CVE", "2016-7117"},
testcases/cve/cve-2017-2671.c +1 -0
index 5d8425248..fb30266eb 100644
@@ -110,6 +110,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_root = 1,
 	.runtime = 40,
+	.min_runtime = 2,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "43a6684519ab"},
 		{"CVE", "2017-2671"},
testcases/kernel/crypto/af_alg07.c +1 -0
index b18d40a6f..b680efbaa 100644
@@ -126,6 +126,7 @@ static struct tst_test test = {
 	.min_kver = "4.10.0",
 	.min_cpus = 2,
 	.runtime = 150,
+	.min_runtime = 2,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "ff7b11aa481f"},
testcases/kernel/pty/pty03.c +1 -0
index 229aa1393..99c659264 100644
@@ -151,6 +151,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_root = 1,
 	.runtime = 30,
+	.min_runtime = 16,
 	.needs_kconfigs = (const char *const[]){
 		"CONFIG_SERIO_SERPORT",
 		NULL
testcases/kernel/pty/pty05.c +1 -0
index 6f6a9bce7..fb76d43b8 100644
@@ -98,6 +98,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.runtime = 150,
+	.min_runtime = 16,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "82f2341c94d27"},
 		{"CVE", "2017-2636"},
testcases/kernel/pty/pty07.c +1 -0
index 1a1c5871b..5d1e26a64 100644
@@ -113,6 +113,7 @@ static struct tst_test test = {
 	.needs_root = 1,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.runtime = 150,
+	.min_runtime = 16,
 	.tags = (const struct tst_tag[]) {
 		{ "linux-git", "6cd1ed50efd8"},
 		{}
testcases/kernel/sound/snd_seq01.c +1 -0
index a23fdb34e..8231e1858 100644
@@ -124,6 +124,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.runtime = 60,
+	.min_runtime = 2,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d15d662e89fc"},
testcases/kernel/sound/snd_timer01.c +1 -0
index bbc5f72a4..53d1ec91b 100644
@@ -137,6 +137,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.runtime = 150,
+	.min_runtime = 2,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d11662f4f798"},
 		{"linux-git", "ba3021b2c79b"},
testcases/kernel/syscalls/bind/bind06.c +1 -1
index 4e7a8d11f..988574a80 100644
@@ -92,7 +92,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 300,
+	.min_runtime = 300,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
testcases/kernel/syscalls/inotify/inotify09.c +1 -0
index 5c9ff3a3b..e61808b69 100644
@@ -96,6 +96,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.test_all = verify_inotify,
 	.runtime = 150,
+	.min_runtime = 2,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d90a10e2444b"},
 		{}
testcases/kernel/syscalls/ipc/shmctl/shmctl05.c +1 -1
index 84d512cfd..6d2415bb9 100644
@@ -104,7 +104,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
-	.runtime = 10,
+	.min_runtime = 40,
 	.setup = setup,
 	.test_all = do_test,
 	.cleanup = cleanup,
testcases/kernel/syscalls/preadv2/preadv203.c +1 -1
index 72a35d3ab..472543e5c 100644
@@ -278,6 +278,6 @@ static struct tst_test test = {
 	.mntpoint = MNTPOINT,
 	.mount_device = 1,
 	.all_filesystems = 1,
-	.runtime = 60,
+	.min_runtime = 60,
 	.needs_root = 1,
 };
testcases/kernel/syscalls/sendmsg/sendmsg03.c +1 -0
index 4a9943a3b..34ebc7e95 100644
@@ -104,6 +104,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.runtime = 150,
+	.min_runtime = 2,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
 		"CONFIG_NET_NS=y",
testcases/kernel/syscalls/setsockopt/setsockopt06.c +1 -1
index 3f266123f..2160b29e7 100644
@@ -108,7 +108,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 270,
+	.min_runtime = 270,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
testcases/kernel/syscalls/setsockopt/setsockopt07.c +1 -1
index ea3dead0f..6eeec8a6f 100644
@@ -121,7 +121,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.runtime = 150,
+	.min_runtime = 150,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
 		"CONFIG_NET_NS=y",
testcases/kernel/syscalls/timerfd/timerfd_settime02.c +1 -0
index 769ff3647..a5a6573e8 100644
@@ -112,6 +112,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.runtime = 150,
+	.min_runtime = 2,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "1e38da300e1e"},
 		{"CVE", "2017-10661"},
testcases/kernel/syscalls/writev/writev03.c +1 -0
index a0b237112..7ef209706 100644
@@ -144,6 +144,7 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.runtime = 75,
+	.min_runtime = 16,
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "d4690f1e1cda"},
 		{}
testcases/network/can/cve/can_bcm01.c +1 -1
index e316a8deb..57ec4989f 100644
@@ -139,7 +139,7 @@ static struct tst_test test = {
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.needs_root = 1,
 	.skip_in_compat = 1,
-	.runtime = 30,
+	.min_runtime = 30,
 	.needs_drivers = (const char *const[]) {
 		"vcan",
 		"can-bcm",
testcases/network/packet/fanout01.c +1 -1
index b2b271aea..dc1ff6ce5 100644
@@ -89,7 +89,7 @@ static struct tst_test test = {
 	.test_all = run,
 	.cleanup = cleanup,
 	.needs_root = 1,
-	.runtime = 180,
+	.min_runtime = 180,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_USER_NS=y",
 		"CONFIG_NET_NS=y",
testcases/network/sockets/vsock01.c +1 -0
index 416bd65d0..e46f663ed 100644
@@ -111,6 +111,7 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
 	.runtime = 60,
+	.min_runtime = 2,
 	.needs_kconfigs = (const char *[]) {
 		"CONFIG_VSOCKETS_LOOPBACK",
 		NULL
Signature · 5 lines
2.49.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author Reviewed-by Wei Gao on v3 1/1
#1 lore raw
Quoted 52 lines · Li Wang

When LTP_RUNTIME_MUL is set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too aggressively. For some tests especially those relying on probabilistic triggers or sampling (e.g., fuzzy sync, CVE stress loops, long fault injection), this can result in too few iterations or missed conditions, leading to unreliable results.

This patch introduces a new field: .min_runtime;

If set, this specifies the minimum allowed runtime (in seconds) after applying the LTP_RUNTIME_MUL scaling factor. The final runtime is calculated as:

MAX(runtime * LTP_RUNTIME_MUL, min_runtime)

If min_runtime is not set, a default minimum of 1 second is enforced.

This approach replaces the need for special flags such as TST_NO_FRAC_RUNTIME_MUL and provides a more systematic, flexible solution.

To validate this logic, I ran the fuzzy_sync tests on a Cortex-A55 CPU at 1.2GHz, running Linux 5.14 (aarch64), with only a single core assigned using taskset to simulate a worst-case, single-core execution scenario:

    time taskset -c 0 ./cve-2016-7117

(The test was locally modified to only perform the sampling phase.)

This setup allows accurate measurement of sampling duration under heavily constrained CPU conditions.

Based on the observed sampling time, I apply the following policy for setting .min_runtime in affected tests:

1. If the sampling time is very short (< 1s), set .min_runtime = 2.

2. If the sampling time is > 2s but still less than .runtime, set .min_runtime = 2 * sampling_time, and leave .runtime unchanged.

3. If the sampling time exceeds .runtime, set .min_runtime = .runtime and remove .runtime.

This ensures that the sampling phase is not prematurely interrupted due to aggressive runtime scaling, which is especially important in CI and low-power platforms.

This change improves test reliability under constrained runtime conditions without sacrificing flexibility for fast-path test execution.

Tested on:
- Cortex-A55, 1.2GHz
- Linux 5.14.0-587.536.el9iv.aarch64

Quoted 5 lines

---

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v3 1/1
#3 lore raw
In reply to Li Wang via ltp

Hi!

lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 633

Quoted 2 lines
lib/tst_test.c +2 -0
+	if (tst_test->min_runtime)
+		min_runtime = MAX(1, tst_test->min_runtime);

Maybe just:

	if (tst_test->min_runtime > 0)
		min_runtime = tst_test->min_runtime;

lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 638

Quoted 5 lines
lib/tst_test.c +1 -1
 	parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
 
-	return runtime * runtime_mul;
+	return MAX(runtime * runtime_mul, min_runtime);
 }

Other than that the approach looks solid to me:

Also sorry for the delay, I was trying to deal with the oldest patches that were being ignored in patchwork so I've ignored the newer patches for a while.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author Applied
#4 lore raw
Quoted 8 lines · Cyril Hrubis

Hi!

lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 633

Quoted 2 lines
lib/tst_test.c +2 -0
+     if (tst_test->min_runtime)
+             min_runtime = MAX(1, tst_test->min_runtime);

Maybe just:

        if (tst_test->min_runtime > 0)
                min_runtime = tst_test->min_runtime;

Yes, the min_runtime type is declared as int so we can avoid using MAX() here.

Quoted 11 lines

lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 638

Quoted 6 lines
      parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);
lib/tst_test.c +1 -1
-     return runtime * runtime_mul;
+     return MAX(runtime * runtime_mul, min_runtime);
 }

Other than that the approach looks solid to me:

Merged, thanks.

From now on, we can safely assign a fractional value to LTP_RUNTIME_MUL.

Quoted 5 lines

Also sorry for the delay, I was trying to deal with the oldest patches that were being ignored in patchwork so I've ignored the newer patches for a while.

Never mind.

Signature · 5 lines
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp