lib: redefine the overall timeout logic of test
RFC PATCH 1/1 lib: redefine the overall timeout logic of test
This patch revises the logic for managing test timeout and runtime limits, introducing a more structured approach. The key changes include renaming max_runtime to runtime for clarity and adding overall_time to represent the total time limit for a test. This ensures better separation of test phases (setup/clenup, runtime) and clearer timeout management.
Key Concepts:
timeout: Represents the time allocated for setup, cleanup, or other
non-runtime parts of the test.
runtime: The maximum runtime of the test's main execution loop. If
exceeded, the test will exit early.
overall_time: The total time allocated for the entire test.
| -- (30s + timeout) * timeout_mul -- | -- runtime * runtime_mul -- |
And, the TST_UNLIMITED_RUNTIME constant has been deprecated and replaced by TST_UNLIMITED_TIMEOUT. This change simplifies the handling of unlimited execution scenarios while aligning the terminology with the new timeout model.
---
Notes:
CI: https://github.com/wangli5665/ltp/actions/runs/12664580981/
doc/old/C-Test-API.asciidoc | 16 ++-- doc/users/setup_tests.rst | 6 +- include/tst_test.h | 28 ++++-- lib/newlib_tests/shell/test_timeout.sh | 10 +- lib/newlib_tests/test_runtime01.c | 2 +- lib/newlib_tests/test_runtime02.c | 2 +- lib/newlib_tests/tst_fuzzy_sync01.c | 2 +- lib/newlib_tests/tst_fuzzy_sync02.c | 2 +- lib/newlib_tests/tst_fuzzy_sync03.c | 2 +- lib/tst_test.c | 95 +++++++++---------- lib/tst_timer_test.c | 2 +- testcases/cve/cve-2014-0196.c | 2 +- testcases/cve/cve-2015-3290.c | 2 +- testcases/cve/cve-2016-7117.c | 2 +- testcases/cve/cve-2017-2671.c | 2 +- .../kernel/controllers/cgroup/cgroup_core03.c | 2 +- testcases/kernel/crypto/af_alg02.c | 2 +- testcases/kernel/crypto/af_alg07.c | 2 +- testcases/kernel/crypto/pcrypt_aead01.c | 2 +- testcases/kernel/fs/fs_fill/fs_fill.c | 2 +- testcases/kernel/fs/fsplough/fsplough.c | 2 +- testcases/kernel/fs/fsx-linux/fsx-linux.c | 2 +- testcases/kernel/fs/read_all/read_all.c | 4 +- testcases/kernel/io/ltp-aiodio/aio-stress.c | 2 +- testcases/kernel/io/ltp-aiodio/aiocp.c | 2 +- .../kernel/io/ltp-aiodio/aiodio_append.c | 2 +- .../kernel/io/ltp-aiodio/aiodio_sparse.c | 2 +- testcases/kernel/io/ltp-aiodio/dio_append.c | 2 +- testcases/kernel/io/ltp-aiodio/dio_read.c | 2 +- testcases/kernel/io/ltp-aiodio/dio_sparse.c | 2 +- testcases/kernel/io/ltp-aiodio/dio_truncate.c | 2 +- testcases/kernel/mem/ksm/ksm02.c | 4 +- testcases/kernel/mem/ksm/ksm04.c | 4 +- .../kernel/mem/mmapstress/mmapstress01.c | 2 +- testcases/kernel/mem/mtest01/mtest01.c | 2 +- testcases/kernel/mem/mtest06/mmap1.c | 2 +- testcases/kernel/mem/mtest06/mmap3.c | 2 +- testcases/kernel/mem/mtest07/mallocstress.c | 2 +- testcases/kernel/mem/oom/oom01.c | 2 +- testcases/kernel/mem/oom/oom02.c | 2 +- testcases/kernel/mem/oom/oom03.c | 2 +- testcases/kernel/mem/oom/oom04.c | 2 +- testcases/kernel/mem/oom/oom05.c | 2 +- testcases/kernel/mem/swapping/swapping01.c | 2 +- testcases/kernel/mem/thp/thp04.c | 2 +- .../kernel/mem/tunable/min_free_kbytes.c | 2 +- testcases/kernel/pty/pty03.c | 2 +- testcases/kernel/pty/pty05.c | 2 +- testcases/kernel/pty/pty06.c | 2 +- testcases/kernel/pty/pty07.c | 2 +- .../sched/cfs-scheduler/cfs_bandwidth01.c | 2 +- .../kernel/sched/cfs-scheduler/starvation.c | 2 +- .../security/dirtyc0w_shmem/dirtyc0w_shmem.c | 2 +- testcases/kernel/security/kallsyms/kallsyms.c | 2 +- testcases/kernel/sound/snd_seq01.c | 2 +- testcases/kernel/sound/snd_timer01.c | 2 +- testcases/kernel/syscalls/bind/bind06.c | 2 +- .../copy_file_range/copy_file_range01.c | 2 +- .../kernel/syscalls/fallocate/fallocate06.c | 2 +- testcases/kernel/syscalls/fork/fork13.c | 2 +- testcases/kernel/syscalls/fsync/fsync02.c | 2 +- .../syscalls/gettimeofday/gettimeofday02.c | 2 +- testcases/kernel/syscalls/inotify/inotify06.c | 2 +- testcases/kernel/syscalls/inotify/inotify09.c | 2 +- testcases/kernel/syscalls/inotify/inotify12.c | 2 +- testcases/kernel/syscalls/ioctl/ioctl_sg01.c | 2 +- .../syscalls/ipc/msgstress/msgstress01.c | 2 +- .../kernel/syscalls/ipc/semget/semget05.c | 2 +- .../kernel/syscalls/ipc/shmctl/shmctl05.c | 2 +- testcases/kernel/syscalls/keyctl/keyctl02.c | 2 +- .../kernel/syscalls/landlock/landlock04.c | 2 +- testcases/kernel/syscalls/madvise/madvise06.c | 2 +- testcases/kernel/syscalls/madvise/madvise11.c | 2 +- .../syscalls/migrate_pages/migrate_pages03.c | 2 +- .../kernel/syscalls/move_pages/move_pages12.c | 2 +- testcases/kernel/syscalls/nice/nice05.c | 2 +- .../perf_event_open/perf_event_open02.c | 2 +- .../perf_event_open/perf_event_open03.c | 2 +- testcases/kernel/syscalls/preadv2/preadv203.c | 2 +- .../kernel/syscalls/readahead/readahead02.c | 8 +- .../syscalls/request_key/request_key03.c | 2 +- .../kernel/syscalls/sendfile/sendfile09.c | 2 +- testcases/kernel/syscalls/sendmsg/sendmsg03.c | 2 +- .../syscalls/set_mempolicy/set_mempolicy01.c | 4 +- .../kernel/syscalls/setsockopt/setsockopt06.c | 2 +- .../kernel/syscalls/setsockopt/setsockopt07.c | 2 +- testcases/kernel/syscalls/swapoff/swapoff01.c | 2 +- testcases/kernel/syscalls/swapon/swapon01.c | 2 +- .../syscalls/timerfd/timerfd_settime02.c | 2 +- testcases/kernel/syscalls/writev/writev03.c | 2 +- testcases/network/can/cve/can_bcm01.c | 2 +- testcases/network/netstress/netstress.c | 2 +- .../network/nfs/nfs_stress/nfs05_make_tree.c | 2 +- testcases/network/packet/fanout01.c | 2 +- testcases/network/sockets/vsock01.c | 2 +- 95 files changed, 178 insertions(+), 171 deletions(-)
doc/old/C-Test-API.asciidoc +8 -8
@@ -95,25 +95,25 @@ in range of [0, '.tcnt' - 1]. IMPORTANT: Only one of '.test' and '.test_all' can be set at a time. Each test has a limit on how long it can run and the limit composes of two -parts max_runtime and timeout. The max_runtime is a limit for how long can the +parts runtime and timeout. The runtime is a limit for how long can the '.test_all' or a set of '.test' functions take and the timeout is static part that should cover the duration of test setup and cleanup plus some safety. Any test that runs for more than a second or two has to make sure to: -- set the runtime either by setting the '.max_runtime' in tst_test or by - calling 'tst_set_max_runtime()' in the test setup +- set the runtime either by setting the '.runtime' in tst_test or by + calling 'tst_set_runtime()' in the test setup - monitor remaning runtime by regular calls to 'tst_remaining_runtime()' and exit when runtime has been used up -Test is free to exit before max_runtime has been used up for example when +Test is free to exit before runtime has been used up for example when minimal number of iteration was finished. The limit is applied to a single call of the '.test_all' function that means that for example when '.test_variants' or '.all_filesystems' is set the whole -test will be limited by 'variants * (max_runtime + timeout)' seconds and the -test runtime will be likely close to 'variants * max_runtime' seconds. +test will be limited by 'variants * (runtime + timeout)' seconds and the +test runtime will be likely close to 'variants * runtime' seconds. [source,c] -------------------------------------------------------------------------------
@@ -401,10 +401,10 @@ WARNING: This function is not thread safe. [source,c] ------------------------------------------------------------------------------- -void tst_set_max_runtime(int max_runtime); +void tst_set_runtime(int runtime); ------------------------------------------------------------------------------- -Allows for setting max_runtime per test iteration dynamically in the test 'setup()', +Allows for setting runtime per test iteration dynamically in the test 'setup()', the timeout is specified in seconds. There are a few testcases whose runtime can vary arbitrarily, these can disable timeouts by setting it to TST_UNLIMITED_RUNTIME.
doc/users/setup_tests.rst +3 -3
@@ -87,14 +87,14 @@ See :master:`testcases/network/README.md`. Test execution time and timeout ------------------------------- -The limit on how long a test can run does compose of two parts: ``max_runtime`` +The limit on how long a test can run does compose of two parts: ``runtime`` and ``timeout``. The limit does apply to a single test variant. That means, for example, that tests which run for all available filesystems will apply this limit for a single filesystem only. -The ``max_runtime`` is a cap on how long the ``run()`` function can take and for +The ``runtime`` is a cap on how long the ``run()`` function can take and for most testcases this part is set to zero. For tests that do run for more than a -second or two the ``max_runtime`` has to be defined and the ``run()`` function +second or two the ``runtime`` has to be defined and the ``run()`` function has to check actively how much runtime is left. Test runtime can be scaled up and down with ``LTP_RUNTIME_MUL`` environment
include/tst_test.h +18 -10
@@ -234,7 +234,7 @@ struct tst_tag {extern unsigned int tst_variant; -#define TST_UNLIMITED_RUNTIME (-1) +#define TST_UNLIMITED_TIMEOUT (-1)/** * struct tst_ulimit_val - An ulimit resource and value.
@@ -435,13 +435,20 @@ struct tst_fs {* The directory is created by the library, the test must not create * it itself. * -* @max_runtime: Maximal test runtime in seconds. Any test that runs for more -* than a second or two should set this and also use -* tst_remaining_runtime() to exit when runtime was used up. -* Tests may finish sooner, for example if requested number of -* iterations was reached before the runtime runs out. If test -* runtime cannot be know in advance it should be set to -* TST_UNLIMITED_RUNTIME. +* @timeout: Maximum allowable time in seconds for test setup and cleanup +* operations. This does not include the runtime of the test itself. +* The timeout ensures that setup or cleanup phases do not hang +* indefinitely and are bounded by a specific duration. If setup +* or cleanup time cannot be predetermined, it should be set to a +* sufficiently large value or TST_UNLIMITED_TIMEOUT for no limit. +* +* @runtime: Maximal test runtime in seconds. Any test that runs for more +* than a second or two should set this and also use +* tst_remaining_runtime() to exit when runtime was used up. +* Tests may finish sooner, for example if requested number of +* iterations was reached before the runtime runs out. If test +* runtime cannot be know in advance it should be set to +* TST_UNLIMITED_RUNTIME. * * @setup: Setup callback is called once at the start of the test in order to * prepare the test environment.
@@ -565,7 +572,8 @@ struct tst_fs {const char *mntpoint; -int max_runtime; +int timeout; +int runtime;void (*setup)(void );void (*cleanup)(void );
@@ -676,7 +684,7 @@ unsigned int tst_remaining_runtime(void);/* * Sets maximal test runtime in seconds. */ -void tst_set_max_runtime (int max_runtime); +void tst_set_runtime (int runtime);/* * Create and open a random file inside the given dir path.
lib/newlib_tests/shell/test_timeout.sh +5 -5
@@ -10,7 +10,7 @@ PATH="$(dirname $0):$(dirname $0)/../../../testcases/lib/:$PATH" # file # timeout_mul # use_cat -# max_runtime +# runtime # expected_exit_code # expected passes # expected failed
@@ -114,8 +114,8 @@ for d in $CLEANED_DATA; do file=$(echo $d | cut -d'|' -f1 | xargs) timeout=$(echo $d | cut -d'|' -f2 | xargs) use_cat=$(echo $d | cut -d'|' -f3 | xargs) - max_runtime=$(echo $d | cut -d'|' -f4 | xargs) - max_runtime=${max_runtime:--1} + runtime=$(echo $d | cut -d'|' -f4 | xargs) + runtime=${runtime:--1} exp_exit=$(echo $d | cut -d'|' -f5 | xargs) exp_exit=${exp_exit:--1} exp_passed=$(echo $d | cut -d'|' -f6 | xargs)
@@ -137,8 +137,8 @@ for d in $CLEANED_DATA; do echo "=== $test_nr/$test_max $description ===" run_test "$file" "$timeout" "$use_cat" - if [ $max_runtime -ne -1 ] && [ $test_duration -gt $max_runtime ]; then - echo "FAILED (runtime: $test_duration, expected less than $max_runtime)" + if [ $runtime -ne -1 ] && [ $test_duration -gt $runtime ]; then + echo "FAILED (runtime: $test_duration, expected less than $runtime)" cur_fails=$((cur_fails + 1)) fi
lib/newlib_tests/test_runtime01.c +1 -1
@@ -25,6 +25,6 @@ static void run(void)static struct tst_test test = { .test_all = run, - .max_runtime =4 , + .runtime =4 , .test_variants =2 , };
lib/newlib_tests/test_runtime02.c +1 -1
@@ -24,5 +24,5 @@ static void run(void)static struct tst_test test = { .test_all = run, - .max_runtime =5 , + .runtime =5 , };
lib/newlib_tests/tst_fuzzy_sync01.c +1 -1
@@ -248,5 +248,5 @@ static struct tst_test test = { .test = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 150, + .runtime = 150, };
lib/newlib_tests/tst_fuzzy_sync02.c +1 -1
@@ -225,5 +225,5 @@ static struct tst_test test = { .test = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 150, + .runtime = 150, };
lib/newlib_tests/tst_fuzzy_sync03.c +1 -1
@@ -99,5 +99,5 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .test_all = run, - .max_runtime = 150, + .runtime = 150, };
lib/tst_test.c +47 -48
184 lines not on this page · raw message
lib/tst_timer_test.c +1 -1
@@ -454,7 +454,7 @@ static void parse_timer_opts(void) runtime_us += tcases[i].usec * tcases[i].samples; } -tst_set_max_runtime ((runtime_us + runtime_us/10 )/1000000 ); +tst_set_runtime ((runtime_us + runtime_us/10 )/1000000 ); }struct tst_test *tst_timer_test_setup (struct tst_test *timer_test)
testcases/cve/cve-2014-0196.c +1 -1
@@ -141,7 +141,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .test_all = run, - .max_runtime = 60, + .runtime = 60, .tags = (const struct tst_tag []) { {"linux-git" ,"4291086b1f08" }, {"CVE" ,"2014-0196" },
testcases/cve/cve-2015-3290.c +1 -1
@@ -519,7 +519,7 @@ static struct tst_test test = { .needs_root = 1, .needs_checkpoints = 1, .setup = setup, - .max_runtime = 180, + .runtime = 180, .test_all = run, .tags = (const struct tst_tag []) { {"linux-git" ,"9b6e6a8334d5" },
testcases/cve/cve-2016-7117.c +1 -1
@@ -149,7 +149,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 60, + .runtime = 60, .tags = (const struct tst_tag []) { {"linux-git" ,"a2e2725541fa" }, {"CVE" ,"2016-7117" },
testcases/cve/cve-2017-2671.c +1 -1
@@ -109,7 +109,7 @@ static struct tst_test test = { .test_all = run, .cleanup = cleanup, .needs_root = 1, - .max_runtime = 40, + .runtime = 40, .tags = (const struct tst_tag []) { {"linux-git" ,"43a6684519ab" }, {"CVE" ,"2017-2671" },
testcases/kernel/controllers/cgroup/cgroup_core03.c +1 -1
@@ -124,7 +124,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .forks_child = 1, - .max_runtime = 20, + .runtime = 20, .needs_cgroup_ctrls = (const char *const []){ "base", NULL }, .needs_cgroup_ver = TST_CG_V2, .needs_checkpoints = 1,
testcases/kernel/crypto/af_alg02.c +1 -1
@@ -77,7 +77,7 @@ static void run(void)static struct tst_test test = { .test_all = run, - .max_runtime =20 , + .runtime =20 , .needs_checkpoints =1 , .tags = (const struct tst_tag []) { {"linux-git" ,"ecaaab564978" },
testcases/kernel/crypto/af_alg07.c +1 -1
@@ -125,7 +125,7 @@ static struct tst_test test = { .cleanup = cleanup, .min_kver = "4.10.0", .min_cpus = 2, - .max_runtime = 150, + .runtime = 150, .taint_check = TST_TAINT_W | TST_TAINT_D, .tags = (const struct tst_tag []) { {"linux-git" ,"ff7b11aa481f" },
testcases/kernel/crypto/pcrypt_aead01.c +1 -1
@@ -75,7 +75,7 @@ static struct tst_test test = { .test_all = run, .cleanup = cleanup, .needs_root = 1, - .max_runtime = 300, + .runtime = 300, .tags = (const struct tst_tag []) { {"linux-git" ,"d76c68109f37" }, {"CVE" ,"2017-18075" },
testcases/kernel/fs/fs_fill/fs_fill.c +1 -1
@@ -121,7 +121,7 @@ static void cleanup(void) }static struct tst_test test = { - .max_runtime =300 , + .runtime =300 , .needs_root =1 , .dev_min_size =1024 , .mount_device =1 ,
testcases/kernel/fs/fsplough/fsplough.c +1 -1
@@ -190,7 +190,7 @@ static void setup(void) MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); filedata =SAFE_MALLOC (filesize); -tst_set_max_runtime (bufsize * loop_count / (8 *1024 *1024 )); +tst_set_runtime (bufsize * loop_count / (8 *1024 *1024 )); }static void run (void )
testcases/kernel/fs/fsx-linux/fsx-linux.c +1 -1
@@ -367,7 +367,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .test_all = run, - .max_runtime = 1800, + .runtime = 1800, .options = (struct tst_option []) { {"l:" , &str_file_max_size,"Maximum size in MB of the test file(s) (default 262144)" }, {"o:" , &str_op_max_size,"Maximum size for single operation (default 65536)" },
testcases/kernel/fs/read_all/read_all.c +2 -2
@@ -598,7 +598,7 @@ static void setup(void) worker_timeout); }else { worker_timeout =10 *tst_remaining_runtime (); -tst_res (TINFO ,"Worker timeout set to 10%% of max_runtime: %dms" , +tst_res (TINFO ,"Worker timeout set to 10%% of runtime: %dms" , worker_timeout); } worker_timeout *=1000 ;
@@ -746,5 +746,5 @@ static struct tst_test test = { .cleanup = cleanup, .test_all = run, .forks_child = 1, - .max_runtime = 100, + .runtime = 100, };
testcases/kernel/io/ltp-aiodio/aio-stress.c +1 -1
@@ -1347,7 +1347,7 @@ static struct tst_test test = { .setup = setup, .needs_tmpdir = 1, .needs_root = 1, - .max_runtime = 1800, + .runtime = 1800, .options = (struct tst_option []){ {"a:" , &str_iterations,"Total number of ayncs I/O the program will run (default 500)" }, {"b:" , &str_max_io_submit,"Max number of iocbs to give io_submit at once" },
testcases/kernel/io/ltp-aiodio/aiocp.c +1 -1
@@ -322,7 +322,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .needs_tmpdir = 1, - .max_runtime = 1800, + .runtime = 1800, .needs_root = 1, .options = (struct tst_option []) { {"b:" , &str_aio_blksize,"Size of writing blocks (default 1K)" },
testcases/kernel/io/ltp-aiodio/aiodio_append.c +1 -1
@@ -190,7 +190,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_tmpdir = 1, .forks_child = 1, - .max_runtime = 1800, + .runtime = 1800, .options = (struct tst_option []) { {"n:" , &str_numchildren,"Number of threads (default 16)" }, {"s:" , &str_writesize,"Size of the file to write (default 64K)" },
testcases/kernel/io/ltp-aiodio/aiodio_sparse.c +1 -1
@@ -244,7 +244,7 @@ static struct tst_test test = {"tmpfs" ,NULL }, - .max_runtime = 1800, + .runtime = 1800, };#else TST_TEST_TCONF ("test requires libaio and its development packages" );
testcases/kernel/io/ltp-aiodio/dio_append.c +1 -1
@@ -96,7 +96,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_tmpdir = 1, .forks_child = 1, - .max_runtime = 1800, + .runtime = 1800, .options = (struct tst_option []) { {"n:" , &str_numchildren,"Number of processes (default 16)" }, {"w:" , &str_writesize,"Write size for each append (default 64K)" },
testcases/kernel/io/ltp-aiodio/dio_read.c +1 -1
@@ -178,7 +178,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_tmpdir = 1, .forks_child = 1, - .max_runtime = 1800, + .runtime = 1800, .options = (struct tst_option []) { {"n:" , &str_numchildren,"Number of threads (default 8)" }, {"w:" , &str_writesize,"Size of writing blocks (default 32M)" },
testcases/kernel/io/ltp-aiodio/dio_sparse.c +1 -1
@@ -135,5 +135,5 @@ static struct tst_test test = { "tmpfs",NULL }, - .max_runtime = 1800, + .runtime = 1800, };
testcases/kernel/io/ltp-aiodio/dio_truncate.c +1 -1
@@ -170,7 +170,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_tmpdir = 1, .forks_child = 1, - .max_runtime = 1800, + .runtime = 1800, .options = (struct tst_option []) { {"n:" , &str_numchildren,"Number of threads (default 16)" }, {"s:" , &str_filesize,"Size of file (default 64K)" },
testcases/kernel/mem/ksm/ksm02.c +2 -2
@@ -87,7 +87,7 @@ static void setup(void)parse_ksm_options (opt_sizestr, &size, opt_numstr, &num, opt_unitstr, &unit);if (opt_sizestr && size >DEFAULT_MEMSIZE ) -tst_set_max_runtime (32 * (size /DEFAULT_MEMSIZE )); +tst_set_runtime (32 * (size /DEFAULT_MEMSIZE )); }static struct tst_test test = {
@@ -116,7 +116,7 @@ static struct tst_test test = {NULL }, .test_all = verify_ksm, - .max_runtime = 32, + .runtime = 32, .needs_cgroup_ctrls = (const char *const []){ "cpuset", NULL }, };
testcases/kernel/mem/ksm/ksm04.c +2 -2
@@ -74,7 +74,7 @@ static void setup(void)SAFE_CG_PRINTF (tst_cg,"cgroup.procs" ,"%d" ,getpid ());if (opt_sizestr && size >DEFAULT_MEMSIZE ) -tst_set_max_runtime (32 * (size /DEFAULT_MEMSIZE )); +tst_set_runtime (32 * (size /DEFAULT_MEMSIZE )); }static struct tst_test test = {
@@ -103,7 +103,7 @@ static struct tst_test test = {NULL }, .test_all = verify_ksm, - .max_runtime = 32, + .runtime = 32, .needs_cgroup_ctrls = (const char *const []){"memory" ,"cpuset" ,NULL },
testcases/kernel/mem/mmapstress/mmapstress01.c +1 -1
@@ -362,7 +362,7 @@ static struct tst_test test = { {}, }, .cleanup = cleanup, - .max_runtime = 12, + .runtime = 12, .needs_tmpdir = 1, .forks_child = 1, };
testcases/kernel/mem/mtest01/mtest01.c +1 -1
@@ -235,7 +235,7 @@ static struct tst_test test = { {"v", &verbose, "Verbose"}, {} }, - .max_runtime = 300, + .runtime = 300, .setup = setup, .cleanup = cleanup, .test_all = mem_test,
testcases/kernel/mem/mtest06/mmap1.c +1 -1
@@ -249,6 +249,6 @@ static void run(void)static struct tst_test test = { .test_all = run, .setup = setup, - .max_runtime =180 , + .runtime =180 , .needs_tmpdir =1 , };
testcases/kernel/mem/mtest06/mmap3.c +1 -1
@@ -182,5 +182,5 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .test_all = test_mmap, - .max_runtime = 60, + .runtime = 60, };
testcases/kernel/mem/mtest07/mallocstress.c +1 -1
@@ -194,7 +194,7 @@ static void cleanup(void) }static struct tst_test test = { - .max_runtime =600 , + .runtime =600 , .needs_checkpoints =1 , .setup = setup, .cleanup = cleanup,
testcases/kernel/mem/oom/oom01.c +1 -1
@@ -36,7 +36,7 @@ static void verify_oom(void)static struct tst_test test = { .needs_root =1 , .forks_child =1 , - .max_runtime =TST_UNLIMITED_RUNTIME , + .timeout =TST_UNLIMITED_TIMEOUT , .test_all = verify_oom, .skip_in_compat =1 , .save_restore = (const struct tst_path_val []) {
testcases/kernel/mem/oom/oom02.c +1 -1
@@ -45,7 +45,7 @@ static void setup(void)static struct tst_test test = { .needs_root =1 , .forks_child =1 , - .max_runtime =TST_UNLIMITED_RUNTIME , + .timeout =TST_UNLIMITED_TIMEOUT , .setup = setup, .test_all = verify_oom, .skip_in_compat =1 ,
testcases/kernel/mem/oom/oom03.c +1 -1
@@ -71,7 +71,7 @@ static void setup(void)static struct tst_test test = { .needs_root =1 , .forks_child =1 , - .max_runtime =TST_UNLIMITED_RUNTIME , + .timeout =TST_UNLIMITED_TIMEOUT , .setup = setup, .test_all = verify_oom, .needs_cgroup_ctrls = (const char *const []){"memory" ,NULL },
testcases/kernel/mem/oom/oom04.c +1 -1
@@ -67,7 +67,7 @@ static void setup(void)static struct tst_test test = { .needs_root =1 , .forks_child =1 , - .max_runtime =TST_UNLIMITED_RUNTIME , + .timeout =TST_UNLIMITED_TIMEOUT , .setup = setup, .test_all = verify_oom, .needs_cgroup_ctrls = (const char *const []){"cpuset" ,NULL },
testcases/kernel/mem/oom/oom05.c +1 -1
@@ -88,7 +88,7 @@ void setup(void)static struct tst_test test = { .needs_root =1 , .forks_child =1 , - .max_runtime =TST_UNLIMITED_RUNTIME , + .timeout =TST_UNLIMITED_TIMEOUT , .setup = setup, .test_all = verify_oom, .needs_cgroup_ctrls = (const char *const []){
testcases/kernel/mem/swapping/swapping01.c +1 -1
@@ -174,7 +174,7 @@ static struct tst_test test = { .needs_root = 1, .forks_child = 1, .min_mem_avail = 10, - .max_runtime = 600, + .runtime = 600, .test_all = test_swapping, .skip_in_compat = 1, .needs_kconfigs = (const char *[]) {
testcases/kernel/mem/thp/thp04.c +1 -1
@@ -161,7 +161,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 150, + .runtime = 150, .tags = (const struct tst_tag []) { {"linux-git" ,"a8f97366452e" }, {"linux-git" ,"8310d48b125d" },
testcases/kernel/mem/tunable/min_free_kbytes.c +1 -1
@@ -221,7 +221,7 @@ static void setup(void)static struct tst_test test = { .needs_root =1 , .forks_child =1 , - .max_runtime =TST_UNLIMITED_RUNTIME , + .timeout =TST_UNLIMITED_TIMEOUT , .setup = setup, .test_all = min_free_kbytes_test, .save_restore = (const struct tst_path_val []) {
testcases/kernel/pty/pty03.c +1 -1
@@ -152,7 +152,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .needs_root = 1, - .max_runtime = 30, + .runtime = 30, .needs_kconfigs = (const char *const []){"CONFIG_SERIO_SERPORT" ,NULL
testcases/kernel/pty/pty05.c +1 -1
@@ -97,7 +97,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, - .max_runtime = 150, + .runtime = 150, .tags = (const struct tst_tag []) { {"linux-git" ,"82f2341c94d27" }, {"CVE" ,"2017-2636" },
testcases/kernel/pty/pty06.c +1 -1
@@ -97,7 +97,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_root = 1, .taint_check = TST_TAINT_W | TST_TAINT_D, - .max_runtime = 150, + .runtime = 150, .tags = (const struct tst_tag []) { {"CVE" ,"2020-36557" }, {"linux-git" ,"ca4463bf8438" },
testcases/kernel/pty/pty07.c +1 -1
@@ -114,7 +114,7 @@ static struct tst_test test = { .cleanup = cleanup, .needs_root = 1, .taint_check = TST_TAINT_W | TST_TAINT_D, - .max_runtime = 150, + .runtime = 150, .tags = (const struct tst_tag []) { {"linux-git" ,"6cd1ed50efd8" }, {}
testcases/kernel/sched/cfs-scheduler/cfs_bandwidth01.c +1 -1
@@ -174,7 +174,7 @@ static struct tst_test test = { .cleanup = cleanup, .forks_child = 1, .needs_checkpoints = 1, - .max_runtime = 20, + .runtime = 20, .taint_check = TST_TAINT_W | TST_TAINT_D, .needs_kconfigs = (const char *[]) {"CONFIG_CFS_BANDWIDTH" ,
testcases/kernel/sched/cfs-scheduler/starvation.c +1 -1
@@ -112,7 +112,7 @@ static void setup(void)if (tst_has_slow_kconfig ())tst_brk (TCONF ,"Skip test due to slow kernel configuration" ); -tst_set_max_runtime (timeout); +tst_set_runtime (timeout); } static void handler(int sig LTP_ATTRIBUTE_UNUSED)
testcases/kernel/security/dirtyc0w_shmem/dirtyc0w_shmem.c +1 -1
@@ -108,7 +108,7 @@ static struct tst_test test = { .needs_checkpoints = 1, .forks_child = 1, .needs_root = 1, - .max_runtime = 120, + .runtime = 120, .setup = setup, .cleanup = cleanup, .test_all = dirtyc0w_shmem_test,
testcases/kernel/security/kallsyms/kallsyms.c +1 -1
@@ -203,7 +203,7 @@ static struct tst_test test = { .needs_root = 1, .setup = setup, .cleanup = cleanup, - .max_runtime = 60, + .runtime = 60, .needs_kconfigs = (const char *const []){"CONFIG_KALLSYMS=y" ,NULL
testcases/kernel/sound/snd_seq01.c +1 -1
@@ -123,7 +123,7 @@ static struct tst_test test = { .tcnt = ARRAY_SIZE(testfunc_list), .setup = setup, .cleanup = cleanup, - .max_runtime = 60, + .runtime = 60, .taint_check = TST_TAINT_W | TST_TAINT_D, .tags = (const struct tst_tag []) { {"linux-git" ,"d15d662e89fc" },
testcases/kernel/sound/snd_timer01.c +1 -1
@@ -136,7 +136,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, - .max_runtime = 150, + .runtime = 150, .tags = (const struct tst_tag []) { {"linux-git" ,"d11662f4f798" }, {"linux-git" ,"ba3021b2c79b" },
testcases/kernel/syscalls/bind/bind06.c +1 -1
@@ -92,7 +92,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 300, + .runtime = 300, .taint_check = TST_TAINT_W | TST_TAINT_D, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,
testcases/kernel/syscalls/copy_file_range/copy_file_range01.c +1 -1
@@ -232,5 +232,5 @@ static struct tst_test test = { .all_filesystems = 1, .test = copy_file_range_verify, .test_variants = TEST_VARIANTS, - .max_runtime = 5 + .runtime = 5 };
testcases/kernel/syscalls/fallocate/fallocate06.c +1 -1
@@ -261,7 +261,7 @@ static struct tst_test test = { .tcnt = ARRAY_SIZE(testcase_list), .needs_root = 1, .dev_min_size = 1024, - .max_runtime = 120, + .runtime = 120, .mount_device = 1, .mntpoint = MNTPOINT, .all_filesystems = 1,
testcases/kernel/syscalls/fork/fork13.c +1 -1
@@ -110,7 +110,7 @@ static void check(void)static struct tst_test test = { .needs_root =1 , .forks_child =1 , - .max_runtime =600 , + .runtime =600 , .test_all = check, .save_restore = (const struct tst_path_val []) { {"/proc/sys/kernel/pid_max" ,PID_MAX_STR ,TST_SR_TBROK },
testcases/kernel/syscalls/fsync/fsync02.c +1 -1
@@ -115,5 +115,5 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .needs_tmpdir = 1, - .max_runtime = 300, + .runtime = 300, };
testcases/kernel/syscalls/gettimeofday/gettimeofday02.c +1 -1
@@ -72,6 +72,6 @@ static void setup(void)static struct tst_test test = { .setup = setup, - .max_runtime =10 , + .runtime =10 , .test_all = verify_gettimeofday, };
testcases/kernel/syscalls/inotify/inotify06.c +1 -1
@@ -116,7 +116,7 @@ static void cleanup(void) }static struct tst_test test = { - .max_runtime =600 , + .runtime =600 , .needs_root =1 , .needs_tmpdir =1 , .forks_child =1 ,
testcases/kernel/syscalls/inotify/inotify09.c +1 -1
@@ -97,7 +97,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .test_all = verify_inotify, - .max_runtime = 150, + .runtime = 150, .tags = (const struct tst_tag []) { {"linux-git" ,"d90a10e2444b" }, {}
testcases/kernel/syscalls/inotify/inotify12.c +1 -1
@@ -153,7 +153,7 @@ static void cleanup(void) }static struct tst_test test = { - .max_runtime =10 , + .runtime =10 , .needs_tmpdir =1 , .cleanup = cleanup, .test = verify_inotify,
testcases/kernel/syscalls/ioctl/ioctl_sg01.c +1 -1
@@ -128,7 +128,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 3600, + .runtime = 3600, .tags = (const struct tst_tag []) { {"linux-git" ,"a45b599ad808" }, {"CVE" ,"2018-1000204" },
testcases/kernel/syscalls/ipc/msgstress/msgstress01.c +1 -1
@@ -311,7 +311,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .forks_child = 1, - .max_runtime = 180, + .runtime = 180, .options = (struct tst_option []) { {"n:" , &str_num_messages,"Number of messages to send (default: 1000)" }, {"l:" , &str_num_iterations,"Number iterations per message (default: "
testcases/kernel/syscalls/ipc/semget/semget05.c +1 -1
@@ -43,7 +43,7 @@ static void setup(void)SAFE_FILE_SCANF ("/proc/sys/kernel/sem" ,"%*d %*d %*d %d" , &maxsems);/* Prevent timeout due to high semaphore array limit */ -tst_set_max_runtime (maxsems /200 ); +tst_set_runtime (maxsems /200 ); sem_id_arr =SAFE_MALLOC ((maxsems - used_cnt) *sizeof (int ));for (num =0 ; num < maxsems - used_cnt; num++) {
testcases/kernel/syscalls/ipc/shmctl/shmctl05.c +1 -1
@@ -106,7 +106,7 @@ static void cleanup(void) }static struct tst_test test = { - .max_runtime =10 , + .runtime =10 , .setup = setup, .test_all = do_test, .cleanup = cleanup,
testcases/kernel/syscalls/keyctl/keyctl02.c +1 -1
@@ -160,7 +160,7 @@ static struct tst_test test = { .needs_root = 1, .setup = setup, .cleanup = cleanup, - .max_runtime = 60, + .runtime = 60, .test_all = do_test, .tags = (const struct tst_tag []) { {"linux-git" ,"b4a1b4f5047e" },
testcases/kernel/syscalls/landlock/landlock04.c +1 -1
@@ -204,5 +204,5 @@ static struct tst_test test = { .mount_device = 1, .mntpoint = SANDBOX_FOLDER, .all_filesystems = 1, - .max_runtime = 360, + .runtime = 360, };
testcases/kernel/syscalls/madvise/madvise06.c +1 -1
@@ -242,7 +242,7 @@ static struct tst_test test = { .setup = setup, .needs_tmpdir = 1, .needs_root = 1, - .max_runtime = 60, + .runtime = 60, .taint_check = TST_TAINT_W | TST_TAINT_D, .save_restore = (const struct tst_path_val []) { {"/proc/sys/vm/swappiness" ,NULL ,
testcases/kernel/syscalls/madvise/madvise11.c +1 -1
@@ -432,7 +432,7 @@ static struct tst_test test = {"CONFIG_MEMORY_FAILURE=y" ,NULL }, - .max_runtime = 30, + .runtime = 30, .needs_checkpoints = 1, .setup = setup, .cleanup = cleanup,
testcases/kernel/syscalls/migrate_pages/migrate_pages03.c +1 -1
@@ -139,7 +139,7 @@ static void migrate_test(void) }static struct tst_test test = { - .max_runtime =300 , + .runtime =300 , .needs_root =1 , .setup = setup, .cleanup = cleanup,
testcases/kernel/syscalls/move_pages/move_pages12.c +1 -1
@@ -342,7 +342,7 @@ static struct tst_test test = { .cleanup = cleanup, .test = do_test, .tcnt = ARRAY_SIZE(tcases), - .max_runtime = 240, + .runtime = 240, .tags = (const struct tst_tag []) { {"linux-git" ,"e66f17ff7177" }, {"linux-git" ,"c9d398fa2378" },
testcases/kernel/syscalls/nice/nice05.c +1 -1
@@ -154,5 +154,5 @@ static struct tst_test test = { .setup = setup, .test_all = verify_nice, .needs_root = 1, - .max_runtime = 3, + .runtime = 3, };
testcases/kernel/syscalls/perf_event_open/perf_event_open02.c +1 -1
@@ -334,5 +334,5 @@ static struct tst_test test = { }, .test_all = verify, .needs_root = 1, - .max_runtime = 72 + .runtime = 72 };
testcases/kernel/syscalls/perf_event_open/perf_event_open03.c +1 -1
@@ -127,7 +127,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .needs_root = 1, - .max_runtime = 300, + .runtime = 300, .tags = (const struct tst_tag []) { {"linux-git" ,"7bdb157cdebb" }, {"CVE" ,"2020-25704" },
testcases/kernel/syscalls/preadv2/preadv203.c +1 -1
@@ -278,6 +278,6 @@ static struct tst_test test = { .mntpoint = MNTPOINT, .mount_device = 1, .all_filesystems = 1, - .max_runtime = 60, + .runtime = 60, .needs_root = 1, };
testcases/kernel/syscalls/readahead/readahead02.c +4 -4
@@ -321,7 +321,7 @@ static void test_readahead(unsigned int n)/* * The time consuming of readahead quite depending on the platform IO -* speed, sometime test timeout when the default max_runtime is used up. +* speed, sometime test timeout when the default runtime is used up. * * readahead02.c:221: TINFO: Test #2: POSIX_FADV_WILLNEED on file * readahead02.c:285: TINFO: read_testfile(0) took: 26317623 usec
@@ -330,7 +330,7 @@ static void test_readahead(unsigned int n)* Here raise the maximum runtime dynamically. */ if ((tc+1 )->readahead) -tst_set_max_runtime (test.max_runtime + (usec + usec_ra) /1000000 ); +tst_set_runtime (test.runtime + (usec + usec_ra) /1000000 ); }
@@ -381,7 +381,7 @@ static void setup(void) {if (opt_fsizestr) { testfile_size =SAFE_STRTOL (opt_fsizestr,1 ,INT_MAX ); -tst_set_max_runtime (1 + testfile_size / (DEFAULT_FILESIZE /32 )); +tst_set_runtime (1 + testfile_size / (DEFAULT_FILESIZE /32 )); }if (access (PROC_IO_FNAME ,F_OK ))
@@ -422,7 +422,7 @@ static struct tst_test test = { }, .test = test_readahead, .tcnt = ARRAY_SIZE(tcases), - .max_runtime = 30, + .runtime = 30, .tags = (const struct tst_tag []) { {"linux-git" ,"b833a3660394" }, {"linux-git" ,"5b910bd615ba" },
testcases/kernel/syscalls/request_key/request_key03.c +1 -1
@@ -215,7 +215,7 @@ static struct tst_test test = { .test = do_test, .tcnt = ARRAY_SIZE(testcase_list), .forks_child = 1, - .max_runtime = 20, + .runtime = 20, .options = (struct tst_option []) { {"b:" , &opt_bug,"Bug to test for (cve-2017-15299 or cve-2017-15951; default is both)" }, {}
testcases/kernel/syscalls/sendfile/sendfile09.c +1 -1
@@ -93,7 +93,7 @@ static struct tst_test test = { .setup = setup, .test = run, .tcnt = ARRAY_SIZE(tc), - .max_runtime = 120, + .runtime = 120, .skip_in_compat = 1, .tags = (const struct tst_tag []) { {"linux-git" ,"5d73320a96fcc" },
testcases/kernel/syscalls/sendmsg/sendmsg03.c +1 -1
@@ -105,7 +105,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, - .max_runtime = 150, + .runtime = 150, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,"CONFIG_NET_NS=y" ,
testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c +2 -2
@@ -43,7 +43,7 @@ static void setup(void)* * Here reset the maximum runtime according to the NUMA nodes. */ -tst_set_max_runtime (test.max_runtime * (1 << nodes->cnt/16 )); +tst_set_runtime (test.runtime * (1 << nodes->cnt/16 )); }static void cleanup (void )
@@ -119,7 +119,7 @@ static struct tst_test test = { .tcnt = 2, .forks_child = 1, .needs_checkpoints = 1, - .max_runtime = 600, + .runtime = 600, };#else
testcases/kernel/syscalls/setsockopt/setsockopt06.c +1 -1
@@ -110,7 +110,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 270, + .runtime = 270, .taint_check = TST_TAINT_W | TST_TAINT_D, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,
testcases/kernel/syscalls/setsockopt/setsockopt07.c +1 -1
@@ -123,7 +123,7 @@ static struct tst_test test = { .test_all = run, .setup = setup, .cleanup = cleanup, - .max_runtime = 150, + .runtime = 150, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,"CONFIG_NET_NS=y" ,
testcases/kernel/syscalls/swapoff/swapoff01.c +1 -1
@@ -53,6 +53,6 @@ static struct tst_test test = { .all_filesystems = 1, .needs_root = 1, .test_all = verify_swapoff, - .max_runtime = 60, + .runtime = 60, .setup = setup };
testcases/kernel/syscalls/swapon/swapon01.c +1 -1
@@ -51,6 +51,6 @@ static struct tst_test test = { .all_filesystems = 1, .needs_cgroup_ctrls = (const char *const []){ "memory", NULL }, .test_all = verify_swapon, - .max_runtime = 60, + .runtime = 60, .setup = setup };
testcases/kernel/syscalls/timerfd/timerfd_settime02.c +1 -1
@@ -111,7 +111,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, - .max_runtime = 150, + .runtime = 150, .tags = (const struct tst_tag []) { {"linux-git" ,"1e38da300e1e" }, {"CVE" ,"2017-10661" },
testcases/kernel/syscalls/writev/writev03.c +1 -1
@@ -145,7 +145,7 @@ static struct tst_test test = { .min_cpus = 2, .setup = setup, .cleanup = cleanup, - .max_runtime = 75, + .runtime = 75, .tags = (const struct tst_tag []) { {"linux-git" ,"d4690f1e1cda" }, {}
testcases/network/can/cve/can_bcm01.c +1 -1
@@ -141,7 +141,7 @@ static struct tst_test test = { .taint_check = TST_TAINT_W | TST_TAINT_D, .needs_root = 1, .skip_in_compat = 1, - .max_runtime = 30, + .runtime = 30, .needs_drivers = (const char *const []) {"vcan" ,"can-bcm" ,
testcases/network/netstress/netstress.c +1 -1
@@ -1024,6 +1024,6 @@ static struct tst_test test = { {"B:", &server_bg, "Run in background, arg is the process directory"}, {} }, - .max_runtime = 300, + .runtime = 300, .needs_checkpoints = 1, };
testcases/network/nfs/nfs_stress/nfs05_make_tree.c +1 -1
@@ -214,5 +214,5 @@ static struct tst_test test = { .options = opts, .test_all = do_test, .setup = setup, - .max_runtime = 300, + .runtime = 300, };
testcases/network/packet/fanout01.c +1 -1
@@ -89,7 +89,7 @@ static struct tst_test test = { .test_all = run, .cleanup = cleanup, .needs_root = 1, - .max_runtime = 180, + .runtime = 180, .needs_kconfigs = (const char *[]) {"CONFIG_USER_NS=y" ,"CONFIG_NET_NS=y" ,
testcases/network/sockets/vsock01.c +1 -1
@@ -112,7 +112,7 @@ static struct tst_test test = { .setup = setup, .cleanup = cleanup, .taint_check = TST_TAINT_W | TST_TAINT_D, - .max_runtime = 60, + .runtime = 60, .needs_kconfigs = (const char *[]) {"CONFIG_VSOCKETS_LOOPBACK" ,NULL
Signature · 5 lines
2.47.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 7 lines
Key Concepts:
timeout: Represents the time allocated for setup, cleanup, or other
non-runtime parts of the test.runtime: The maximum runtime of the test's main execution loop. If
exceeded, the test will exit early.
This is not entirely correct. I guess that we still didn't understand each other. Let me try to explain once more.
The runtime is supposed to be used only in tests that call tst_remaning_runtime() in the main loop to limit the test duration based on runtime. This means that the test function execution time is always the same (plus minus some rounding errors).
And the timeout is supposed to be upper bound for everything in the test minus the part that is limited by the runtime.
So for the tests that use tst_remaining_runtime() the timeout guards the setup and cleanup, but for tests that does not use tst_remaining_runtime() the timeout limits the whole test run.
The reasoning behind this is that if for example we have a test where the test function does I/O that is known to take up to 10s we want to multiply this in the case of the debug kernel because the test function may be slower.
On the other hand test function that has a loop that checks the remaning runtime will do less iterations on debug kernel but the duration will be the same. And we do not want to multiply this value because it will make the test even slower on debug kernels.
The changes mostly looks okay, but some of the tests below that had .max_runtime should replace that with .timeout instead. And we also need tst_set_timeout() function for the testcases that try to figure out the upper bound for the test duration in the test setup.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 17 lines · Cyril Hrubis
Hi!
Quoted 7 lines
Key Concepts:
timeout: Represents the time allocated for setup, cleanup, or other
non-runtime parts of the test.runtime: The maximum runtime of the test's main execution loop. If
exceeded, the test will exit early.This is not entirely correct. I guess that we still didn't understand each other. Let me try to explain once more.
The runtime is supposed to be used only in tests that call
tst_remaning_runtime()in the main loop to limit the test duration based on runtime. This means that the test function execution time is always the same (plus minus some rounding errors).
Quoted 7 lines
And the timeout is supposed to be upper bound for everything in the test minus the part that is limited by the runtime.
So for the tests that use
tst_remaining_runtime()the timeout guards the setup and cleanup, but for tests that does not usetst_remaining_runtime()the timeout limits the whole test run.
Ok, I see the difference. Thanks for explaining again.
Quoted 10 lines
The reasoning behind this is that if for example we have a test where the test function does I/O that is known to take up to 10s we want to multiply this in the case of the debug kernel because the test function may be slower.
On the other hand test function that has a loop that checks the remaning runtime will do less iterations on debug kernel but the duration will be the same. And we do not want to multiply this value because it will make the test even slower on debug kernels.
Indeed.
Quoted 4 lines
The changes mostly looks okay, but some of the tests below that had .max_runtime should replace that with .timeout instead. And we also
Btw, we have some tests that implicitly invoke tst_remaining_runtime, for example, using the fuzzy_sync library, which also needs .runtime but not .timeout.
need tst_set_timeout() function for the testcases that try to figure out
Quoted 2 lines
the upper bound for the test duration in the test setup.
+1, I will add this function and comment on the details of timeout and runtime in LTP doc/.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 9 lines · Li Wang
Quoted 4 lines
The changes mostly looks okay, but some of the tests below that had .max_runtime should replace that with .timeout instead. And we also
Btw, we have some tests that implicitly invoke
tst_remaining_runtime, for example, using thefuzzy_synclibrary, which also needs .runtime but not .timeout.
Also, tests that use 'test.runtime' directly (e.g. readahead02.c, set_mempolicy01.c)
must continue to be marked as using .runtime.
Quoted 1 line
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 8 lines
Quoted 4 lines
Btw, we have some tests that implicitly invoke
tst_remaining_runtime, for example, using thefuzzy_synclibrary, which also needs .runtime but not .timeout.Also, tests that use 'test.runtime' directly (e.g.
readahead02.c,set_mempolicy01.c)
must continue to be marked as using .runtime.
I think that readahead02 is a case where we should switch to timeout because that is exactly the situation where want the timeout to be multiplied when the system is slower. The way the test adjust the runtime dynamically is wrong anyways, because it increases the timeout for each iteration. We should just put the worstcase runtime into the .timeout instead.
And similarily for the the mempolicy it looks like we should set the timeout dynamicaly in the testup with tst_set_timeout() instead of runtime.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 10 lines · Cyril Hrubis
Hi!
Quoted 8 lines
Quoted 4 lines
Btw, we have some tests that implicitly invoke
tst_remaining_runtime, for example, using thefuzzy_synclibrary, which also needs .runtime but not .timeout.Also, tests that use 'test.runtime' directly (e.g.
readahead02.c,set_mempolicy01.c)
must continue to be marked as using .runtime.
Quoted 7 lines
I think that readahead02 is a case where we should switch to timeout because that is exactly the situation where want the timeout to be multiplied when the system is slower. The way the test adjust the runtime dynamically is wrong anyways, because it increases the timeout for each iteration. We should just put the worstcase runtime into the .timeout instead.
I have a different view on the readahead02 test, because the runtime resetting is based on pieces of each IO test elapsed time, then reset runtime for next time. This applies to any kernels, no matter the faster or slower, the elapsed time will be enough for the next tcases[].
If we put the worst-case runtime into .timeout and reset for the next tcases[], which will be multiplied again on debug kernel, actually we don't need that since the dynamic runtime comes from a real test.
Maybe I missed something in the test, but we can treat readahead02 in a separate work. The new patch 4/4 should be modified using .runtime instead of .timeout. Feel free to comment your thoughts there.
Quoted 5 lines
And similarily for the the mempolicy it looks like we should set the timeout dynamicaly in the testup with
tst_set_timeout()instead of runtime.
Agreed this change on mempolicy01.c.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 8 lines
I have a different view on the readahead02 test, because the runtime resetting is based on pieces of each IO test elapsed time, then reset runtime for next time. This applies to any kernels, no matter the faster or slower, the elapsed time will be enough for the next tcases[].
If we put the worst-case runtime into .timeout and reset for the next tcases[], which will be multiplied again on debug kernel, actually we don't need that since the dynamic runtime comes from a real test.
I do not get why we need to reset anything. The test library does reset the timeout after each test iteration:
for (i = 0; i < tst_test->tcnt; i++) {
saved_results = *results;
heartbeat();
^
this resets the timeout timer so each iteration runs
with the whole timeout
tst_test->test(i);
if (tst_getpid() != main_pid)
exit(0);
tst_reap_children();
if (results_equal(&saved_results, results))
tst_brk(TBROK, "Test %i haven't reported results!", i);
}Quoted 3 lines
Maybe I missed something in the test, but we can treat readahead02 in a separate work. The new patch 4/4 should be modified using .runtime instead of .timeout. Feel free to comment your thoughts there.
Why is readahead02 so special that we need to do more than the usuall?
That is:
- measure how long one tests iteration takes on slower hardware
- multiply that by two and set that as a .timeout
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 23 lines · Cyril Hrubis
Hi!
Quoted 8 lines
I have a different view on the readahead02 test, because the runtime resetting is based on pieces of each IO test elapsed time, then reset runtime for next time. This applies to any kernels, no matter the faster or slower, the elapsed time will be enough for the next tcases[].
If we put the worst-case runtime into .timeout and reset for the next tcases[], which will be multiplied again on debug kernel, actually we don't need that since the dynamic runtime comes from a real test.
I do not get why we need to reset anything. The test library does reset the timeout after each test iteration:
for (i = 0; i < tst_test->tcnt; i++) { saved_results = *results; heartbeat(); ^ this resets the timeout timer so each iteration runs with the whole timeout tst_test->test(i); if (tst_getpid() != main_pid)
Quoted 9 lines
exit(0);
tst_reap_children();
if (results_equal(&saved_results, results))
tst_brk(TBROK, "Test %i haven't reported
results!", i);
}Yes, that's right.
Quoted 11 lines
Quoted 3 lines
Maybe I missed something in the test, but we can treat readahead02 in a separate work. The new patch 4/4 should be modified using .runtime instead of .timeout. Feel free to comment your thoughts there.
Why is readahead02 so special that we need to do more than the usuall?
That is:
- measure how long one tests iteration takes on slower hardware
- multiply that by two and set that as a .timeout
I admit that this is the recommended/standard way to use .timeout based on the new time logic design.
P.s.
The reason for calling tst_set_runtime() in each iteration is that we can directly utilize the real elapsed time as runtime and don't need to do additional measurements.
That value (which comes from a real test) is more precise than (what we get on) any other slower hardware. And the disadvantage as you pointed it increases the timeout for each iteration.
I can give up this usage and follow your suggestion, but I still fell that we could have one more way on .runtime.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 52 lines · Li Wang
Quoted 23 lines · Cyril Hrubis
Hi!
Quoted 8 lines
I have a different view on the readahead02 test, because the runtime resetting is based on pieces of each IO test elapsed time, then reset runtime for next time. This applies to any kernels, no matter the faster or slower, the elapsed time will be enough for the next tcases[].
If we put the worst-case runtime into .timeout and reset for the next tcases[], which will be multiplied again on debug kernel, actually we don't need that since the dynamic runtime comes from a real test.
I do not get why we need to reset anything. The test library does reset the timeout after each test iteration:
for (i = 0; i < tst_test->tcnt; i++) { saved_results = *results; heartbeat(); ^ this resets the timeout timer so each iteration runs with the whole timeout tst_test->test(i); if (tst_getpid() != main_pid)Quoted 9 lines
exit(0); tst_reap_children(); if (results_equal(&saved_results, results)) tst_brk(TBROK, "Test %i haven't reported results!", i); }Yes, that's right.
Quoted 11 lines
Quoted 3 lines
Maybe I missed something in the test, but we can treat readahead02 in a separate work. The new patch 4/4 should be modified using .runtime instead of .timeout. Feel free to comment your thoughts there.
Why is readahead02 so special that we need to do more than the usuall?
That is:
- measure how long one tests iteration takes on slower hardware
- multiply that by two and set that as a .timeout
Quoted 9 lines
I admit that this is the recommended/standard way to use .timeout based on the new time logic design.
P.s.
The reason for calling
tst_set_runtime()in each iteration is that we can directly utilize the real elapsed time as runtime and don't need to do additional measurements.
Forgot to say, that each iteration has four sub-tcases to accomplish. The recommended way (you gave above) is to regard them as one big test and reset timeout by heartbeat() in tst_test->tcnt. If this value is big enough, that's fine. But I fear that value (come from measurement) is still not covered all situations.
Quoted 13 lines
That value (which comes from a real test) is more precise than (what we get on) any other slower hardware. And the disadvantage as you pointed it increases the timeout for each iteration.
I can give up this usage and follow your suggestion, but I still fell that we could have one more way on .runtime.
--
Regards,
Li Wang
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 10 lines
Quoted 4 lines
The reason for calling
tst_set_runtime()in each iteration is that we can directly utilize the real elapsed time as runtime and don't need to do additional measurements.Forgot to say, that each iteration has four sub-tcases to accomplish. The recommended way (you gave above) is to regard them as one big test and reset timeout by
heartbeat()intst_test->tcnt. If this value is big enough, that's fine. But I fear that value (come from measurement) is still not covered all situations.
I agree that the test is a bit unpredictable, as it synces filesytem and drops caches twice in each iteration. However the .timeout is a safety mechanism, so nothing stops us for making it 4x or 8x of the usuall test execution time on slower hardware.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 10 lines · Cyril Hrubis
Hi!
Quoted 1 line
Quoted 1 line
The reason for calling
tst_set_runtime()in each iteration is that wecan
Quoted 7 lines
Quoted 3 lines
directly utilize the real elapsed time as runtime and don't need to do additional measurements.
Forgot to say, that each iteration has four sub-tcases to accomplish. The recommended way (you gave above) is to regard them as one big test and reset timeout by
heartbeat()intst_test->tcnt. If this
Quoted 9 lines
value
Quoted 2 lines
is big enough, that's fine. But I fear that value (come from measurement) is still not covered all situations.
I agree that the test is a bit unpredictable, as it synces filesytem and drops caches twice in each iteration. However the .timeout is a safety mechanism, so nothing stops us for making it 4x or 8x of the usuall test execution time on slower hardware.
Ok, I will collect a proper timeout value on my RPi4 and go with the recommendation way.
Then, let's see how it goes in the future.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp