tst_test: Add min_runtime to control lower bound of scaled runtime
PATCH v3 1/1 tst_test: Add min_runtime to control lower bound of scaled runtime
When LTP_RUNTIME_MUL is set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too aggressively. For some tests especially those relying on probabilistic triggers or sampling (e.g., fuzzy sync, CVE stress loops, long fault injection), this can result in too few iterations or missed conditions, leading to unreliable results.
This patch introduces a new field: .min_runtime;
If set, this specifies the minimum allowed runtime (in seconds) after applying the LTP_RUNTIME_MUL scaling factor. The final runtime is calculated as:
MAX(runtime * LTP_RUNTIME_MUL, min_runtime)
If min_runtime is not set, a default minimum of 1 second is enforced.
This approach replaces the need for special flags such as TST_NO_FRAC_RUNTIME_MUL and provides a more systematic, flexible solution.
To validate this logic, I ran the fuzzy_sync tests on a Cortex-A55 CPU at 1.2GHz, running Linux 5.14 (aarch64), with only a single core assigned using taskset to simulate a worst-case, single-core execution scenario:
time taskset -c 0 ./cve-2016-7117
(The test was locally modified to only perform the sampling phase.)
This setup allows accurate measurement of sampling duration under heavily constrained CPU conditions.
Based on the observed sampling time, I apply the following policy for setting .min_runtime in affected tests:
1. If the sampling time is very short (< 1s), set .min_runtime = 2.
2. If the sampling time is > 2s but still less than .runtime, set .min_runtime = 2 * sampling_time, and leave .runtime unchanged.
3. If the sampling time exceeds .runtime, set .min_runtime = .runtime and remove .runtime.
This ensures that the sampling phase is not prematurely interrupted due to aggressive runtime scaling, which is especially important in CI and low-power platforms.
This change improves test reliability under constrained runtime conditions without sacrificing flexibility for fast-path test execution.
Tested on:
- Cortex-A55, 1.2GHz
- Linux 5.14.0-587.536.el9iv.aarch64
--- include/tst_test.h | 9 +++++++++ lib/tst_test.c | 11 +++++++++-- testcases/cve/cve-2014-0196.c | 2 +- testcases/cve/cve-2016-7117.c | 1 + testcases/cve/cve-2017-2671.c | 1 + testcases/kernel/crypto/af_alg07.c | 1 + testcases/kernel/pty/pty03.c | 1 + testcases/kernel/pty/pty05.c | 1 + testcases/kernel/pty/pty07.c | 1 + testcases/kernel/sound/snd_seq01.c | 1 + testcases/kernel/sound/snd_timer01.c | 1 + testcases/kernel/syscalls/bind/bind06.c | 2 +- testcases/kernel/syscalls/inotify/inotify09.c | 1 + testcases/kernel/syscalls/ipc/shmctl/shmctl05.c | 2 +- testcases/kernel/syscalls/preadv2/preadv203.c | 2 +- testcases/kernel/syscalls/sendmsg/sendmsg03.c | 1 + testcases/kernel/syscalls/setsockopt/setsockopt06.c | 2 +- testcases/kernel/syscalls/setsockopt/setsockopt07.c | 2 +- testcases/kernel/syscalls/timerfd/timerfd_settime02.c | 1 + testcases/kernel/syscalls/writev/writev03.c | 1 + testcases/network/can/cve/can_bcm01.c | 2 +- testcases/network/packet/fanout01.c | 2 +- testcases/network/sockets/vsock01.c | 1 + 23 files changed, 39 insertions(+), 10 deletions(-)
include/tst_test.h +9 -0
@@ -460,6 +460,14 @@ struct tst_fs {* (e.g., TIMEOUT_MUL), ensuring consistent test duration across * different environments (e.g., debug vs. stock kernels). * +* @min_runtime: Optional lower bound (in seconds) applied after runtime is +* scaled by LTP_RUNTIME_MUL. If the scaled runtime is smaller +* than this value, it will be clamped up to min_runtime. +* This is useful for tests that require a minimum execution +* time to gather sufficient samples or trigger events (e.g., +* probabilistic or fuzzy synchronization tests). +* If not set, a default minimum of 1 second is enforced. +* * @setup: Setup callback is called once at the start of the test in order to * prepare the test environment. *
@@ -584,6 +592,7 @@ struct tst_fs {int timeout;int runtime; +int min_runtime;void (*setup)(void );void (*cleanup)(void );
lib/tst_test.c +9 -2
@@ -627,10 +627,14 @@ static void parse_mul(float *mul, const char *env_name, float min, float max)static int multiply_runtime (unsigned int runtime) {static float runtime_mul =-1 ; +int min_runtime =1 ; + +if (tst_test->min_runtime) + min_runtime =MAX (1 , tst_test->min_runtime);parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -return runtime * runtime_mul; +return MAX (runtime * runtime_mul, min_runtime); } static struct option {
@@ -662,7 +666,7 @@ static void print_help(void)fprintf (stderr,"LTP_SINGLE_FS_TYPE Specifies filesystem instead all supported (for .all_filesystems)\n" );fprintf (stderr,"LTP_FORCE_SINGLE_FS_TYPE Testing only. The same as LTP_SINGLE_FS_TYPE but ignores test skiplist.\n" );fprintf (stderr,"LTP_TIMEOUT_MUL Timeout multiplier (must be a number >=1)\n" ); -fprintf (stderr,"LTP_RUNTIME_MUL Runtime multiplier (must be a number >=1)\n" ); +fprintf (stderr,"LTP_RUNTIME_MUL Runtime multiplier (must be a number >0)\n" );fprintf (stderr,"LTP_VIRT_OVERRIDE Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n" );fprintf (stderr,"TMPDIR Base directory for template directory (for .needs_tmpdir, default: %s)\n" ,TEMPDIR );fprintf (stderr,"\n" );
@@ -1996,6 +2000,9 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)uname (&uval);tst_res (TINFO ,"Tested kernel: %s %s %s" , uval.release, uval.version, uval.machine); +if (tst_test->min_runtime && !tst_test->runtime) + tst_test->runtime = tst_test->min_runtime; +if (tst_test->runtime) context->runtime =multiply_runtime (tst_test->runtime);
testcases/cve/cve-2014-0196.c +1 -1
@@ -144,7 +144,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .test_all = run, - .runtime = 60, + .min_runtime = 60, .tags = (const struct tst_tag []) { {"linux-git" ,"4291086b1f08" }, {"CVE" ,"2014-0196" },
testcases/cve/cve-2016-7117.c +1 -0
@@ -152,6 +152,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .runtime = 60, + .min_runtime = 12, .tags = (const struct tst_tag []) { {"linux-git" ,"34b88a68f26a" }, {"CVE" ,"2016-7117" },
testcases/cve/cve-2017-2671.c +1 -0
@@ -110,6 +110,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_root = 1, .runtime = 40, + .min_runtime = 2, .tags = (const struct tst_tag []) { {"linux-git" ,"43a6684519ab" }, {"CVE" ,"2017-2671" },
testcases/kernel/crypto/af_alg07.c +1 -0
@@ -126,6 +126,7 @@ static struct tst_test test = { .min_kver = "4.10.0", .min_cpus = 2, .runtime = 150, + .min_runtime = 2, .taint_check = TST_TAINT_W | TST_TAINT_D, .tags = (const struct tst_tag []) { {"linux-git" ,"ff7b11aa481f" },
testcases/kernel/pty/pty03.c +1 -0
@@ -151,6 +151,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_root = 1, .runtime = 30, + .min_runtime = 16, .needs_kconfigs = (const char *const []){"CONFIG_SERIO_SERPORT" ,NULL
testcases/kernel/pty/pty05.c +1 -0
@@ -98,6 +98,7 @@ static struct tst_test test = { .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, .runtime = 150, + .min_runtime = 16, .tags = (const struct tst_tag []) { {"linux-git" ,"82f2341c94d27" }, {"CVE" ,"2017-2636" },
testcases/kernel/pty/pty07.c +1 -0
@@ -113,6 +113,7 @@ static struct tst_test test = { .needs_root = 1, .taint_check = TST_TAINT_W | TST_TAINT_D, .runtime = 150, + .min_runtime = 16, .tags = (const struct tst_tag []) { {"linux-git" ,"6cd1ed50efd8" }, {}
testcases/kernel/sound/snd_seq01.c +1 -0
@@ -124,6 +124,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .runtime = 60, + .min_runtime = 2, .taint_check = TST_TAINT_W | TST_TAINT_D, .tags = (const struct tst_tag []) { {"linux-git" ,"d15d662e89fc" },
testcases/kernel/sound/snd_timer01.c +1 -0
@@ -137,6 +137,7 @@ static struct tst_test test = { .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, .runtime = 150, + .min_runtime = 2, .tags = (const struct tst_tag []) { {"linux-git" ,"d11662f4f798" }, {"linux-git" ,"ba3021b2c79b" },
testcases/kernel/syscalls/bind/bind06.c +1 -1
@@ -92,7 +92,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .runtime = 300, + .min_runtime = 300, .taint_check = TST_TAINT_W | TST_TAINT_D, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,
testcases/kernel/syscalls/inotify/inotify09.c +1 -0
@@ -96,6 +96,7 @@ static struct tst_test test = { .cleanup = cleanup, .test_all = verify_inotify, .runtime = 150, + .min_runtime = 2, .tags = (const struct tst_tag []) { {"linux-git" ,"d90a10e2444b" }, {}
testcases/kernel/syscalls/ipc/shmctl/shmctl05.c +1 -1
@@ -104,7 +104,7 @@ static void cleanup(void) }static struct tst_test test = { - .runtime =10 , + .min_runtime =40 , .setup = setup, .test_all = do_test, .cleanup = cleanup,
testcases/kernel/syscalls/preadv2/preadv203.c +1 -1
@@ -278,6 +278,6 @@ static struct tst_test test = { .mntpoint = MNTPOINT, .mount_device = 1, .all_filesystems = 1, - .runtime = 60, + .min_runtime = 60, .needs_root = 1, };
testcases/kernel/syscalls/sendmsg/sendmsg03.c +1 -0
@@ -104,6 +104,7 @@ static struct tst_test test = { .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, .runtime = 150, + .min_runtime = 2, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,"CONFIG_NET_NS=y" ,
testcases/kernel/syscalls/setsockopt/setsockopt06.c +1 -1
@@ -108,7 +108,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .runtime = 270, + .min_runtime = 270, .taint_check = TST_TAINT_W | TST_TAINT_D, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,
testcases/kernel/syscalls/setsockopt/setsockopt07.c +1 -1
@@ -121,7 +121,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .runtime = 150, + .min_runtime = 150, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,"CONFIG_NET_NS=y" ,
testcases/kernel/syscalls/timerfd/timerfd_settime02.c +1 -0
@@ -112,6 +112,7 @@ static struct tst_test test = { .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, .runtime = 150, + .min_runtime = 2, .tags = (const struct tst_tag []) { {"linux-git" ,"1e38da300e1e" }, {"CVE" ,"2017-10661" },
testcases/kernel/syscalls/writev/writev03.c +1 -0
@@ -144,6 +144,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .runtime = 75, + .min_runtime = 16, .tags = (const struct tst_tag []) { {"linux-git" ,"d4690f1e1cda" }, {}
testcases/network/can/cve/can_bcm01.c +1 -1
@@ -139,7 +139,7 @@ static struct tst_test test = { .taint_check = TST_TAINT_W | TST_TAINT_D, .needs_root = 1, .skip_in_compat = 1, - .runtime = 30, + .min_runtime = 30, .needs_drivers = (const char *const []) {"vcan" ,"can-bcm" ,
testcases/network/packet/fanout01.c +1 -1
@@ -89,7 +89,7 @@ static struct tst_test test = { .test_all = run, .cleanup = cleanup, .needs_root = 1, - .runtime = 180, + .min_runtime = 180, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,"CONFIG_NET_NS=y" ,
testcases/network/sockets/vsock01.c +1 -0
@@ -111,6 +111,7 @@ static struct tst_test test = { .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, .runtime = 60, + .min_runtime = 2, .needs_kconfigs = (const char *[]) {"CONFIG_VSOCKETS_LOOPBACK" ,NULL
Signature · 5 lines
2.49.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 52 lines · Li Wang
When
LTP_RUNTIME_MULis set to a value < 1.0 (commonly in CI or fast test modes), the effective runtime of tests may be scaled down too aggressively. For some tests especially those relying on probabilistic triggers or sampling (e.g., fuzzy sync, CVE stress loops, long fault injection), this can result in too few iterations or missed conditions, leading to unreliable results.This patch introduces a new field: .min_runtime;
If set, this specifies the minimum allowed runtime (in seconds) after applying the
LTP_RUNTIME_MULscaling factor. The final runtime is calculated as:
MAX(runtime * LTP_RUNTIME_MUL, min_runtime)If
min_runtimeis not set, a default minimum of 1 second is enforced.This approach replaces the need for special flags such as
TST_NO_FRAC_RUNTIME_MULand provides a more systematic, flexible solution.To validate this logic, I ran the
fuzzy_synctests on a Cortex-A55 CPU at 1.2GHz, running Linux 5.14 (aarch64), with only a single core assigned usingtasksetto simulate a worst-case, single-core execution scenario:time taskset -c 0 ./cve-2016-7117(The test was locally modified to only perform the sampling phase.)
This setup allows accurate measurement of sampling duration under heavily constrained CPU conditions.
Based on the observed sampling time, I apply the following policy for setting
.min_runtimein affected tests:1. If the sampling time is very short (< 1s), set .min_runtime = 2.
2. If the sampling time is > 2s but still less than .runtime, set .min_runtime = 2 *
sampling_time, and leave .runtime unchanged.3. If the sampling time exceeds .runtime, set .min_runtime = .runtime and remove .runtime.
This ensures that the sampling phase is not prematurely interrupted due to aggressive runtime scaling, which is especially important in CI and low-power platforms.
This change improves test reliability under constrained runtime conditions without sacrificing flexibility for fast-path test execution.
Tested on:
- Cortex-A55, 1.2GHz
- Linux 5.14.0-587.536.el9iv.aarch64
Quoted 5 lines
Suggested-by Martin Doucha <[email protected]>Signed-off-by Li Wang <[email protected]>Cc Ian Wienand <[email protected]>Cc Wei Gao <[email protected]>---
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 2 lines
Reviewed-by Wei Gao <[email protected]>
Thanks Wei for the review!
@Martin Doucha <[email protected]> @Cyril Hrubis <[email protected]> @Petr Vorel <[email protected]>
Hi Martin and All, do you have any more comments on this final patch?
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 633
Quoted 2 lines
lib/tst_test.c +2
-0+if (tst_test->min_runtime) + min_runtime =MAX (1 , tst_test->min_runtime);
Maybe just:
if (tst_test->min_runtime > 0) min_runtime = tst_test->min_runtime;
lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 638
Quoted 5 lines
lib/tst_test.c +1
-1parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -return runtime * runtime_mul; +return MAX (runtime * runtime_mul, min_runtime); }
Other than that the approach looks solid to me:
Also sorry for the delay, I was trying to deal with the oldest patches that were being ignored in patchwork so I've ignored the newer patches for a while.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Li Wang via ltp
AppliedQuoted 8 lines · Cyril Hrubis
Hi!
lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 633
Quoted 2 lines
lib/tst_test.c +2
-0+if (tst_test->min_runtime) + min_runtime =MAX (1 , tst_test->min_runtime);Maybe just:
if (tst_test->min_runtime > 0) min_runtime = tst_test->min_runtime;
Yes, the min_runtime type is declared as int so we can avoid using MAX() here.
Quoted 11 lines
lib/tst_test.c static void parse_mul(float *mul, const char *env_name, float min, float max) · line 638
Quoted 6 lines
parse_mul(&runtime_mul, "LTP_RUNTIME_MUL", 0.0099, 100);lib/tst_test.c +1
-1-return runtime * runtime_mul; +return MAX (runtime * runtime_mul, min_runtime); }Other than that the approach looks solid to me:
Reviewed-by Cyril Hrubis <[email protected]>
Merged, thanks.
From now on, we can safely assign a fractional value to LTP_RUNTIME_MUL.
Quoted 5 lines
Also sorry for the delay, I was trying to deal with the oldest patches that were being ignored in patchwork so I've ignored the newer patches for a while.
Never mind.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp