lib: multiply the timeout if detect slow kconfigs
PATCH 1/2 lib: multiply the timeout if detect slow kconfigs
This refines the handling of timeouts for tests running on systems with slow kernel configurations (kconfigs).
Previously, the max_runtime was multiplied globally when slow kconfigs were detected, which inadvertently prolonged the runtime of all tests using max_runtime for control.
This patch corrects that behavior by applying the multiplication specifically to timeouts, ensuring it only affects the intended operations without impacting other tests.
--- lib/tst_test.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
lib/tst_test.c +3 -3
@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }
@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned int timeout)if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ; +return timeout * timeout_mul; }
Signature · 5 lines
2.47.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH 2/2 starvation: skip test on slow kernel
Systems with slow kernel configurations may not meet the performance requirements necessary for the starvation test to produce valid results.
Skipping the test ensures that it runs only on systems where its results are meaningful.
--- testcases/kernel/sched/cfs-scheduler/starvation.c | 4 ++++ 1 file changed, 4 insertions(+)
testcases/kernel/sched/cfs-scheduler/starvation.c +4 -0
@@ -21,6 +21,7 @@#include <sched.h> #include "tst_test.h" +#include "tst_kconfig.h" #include "tst_safe_clocks.h" #include "tst_timer.h"
@@ -108,6 +109,9 @@ static void setup(void)else timeout =callibrate () /1000 ; +if (tst_has_slow_kconfig ()) +tst_brk (TCONF ,"Skip test due to slow kernel configuration" ); +tst_set_max_runtime (timeout); }
Signature · 5 lines
2.47.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 2/2] starvation: skip test on slow kernel
Hi Li,
Quoted 6 lines
Systems with slow kernel configurations may not meet the performance requirements necessary for the starvation test to produce valid results.
Skipping the test ensures that it runs only on systems where its results are meaningful.
...
testcases/kernel/sched/cfs-scheduler/starvation.c static void setup(void) · line 114
Quoted 3 lines
testcases/kernel/sched/cfs-scheduler/starvation.c +3
-0+if (tst_has_slow_kconfig ()) +tst_brk (TCONF ,"Skip test due to slow kernel configuration" ); +
I suppose you have strong reason for this (e.g. it's failing on some slow machine). I'm testing it on emulated riscv64 to see whether it's needed to be skipped. Because due CONFIG_LATENCYTOP it will always be skipped on Tumbleweed.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 2/2] starvation: skip test on slow kernel
Quoted 15 lines
Hi Li,
Quoted 6 lines
Systems with slow kernel configurations may not meet the performance requirements necessary for the starvation test to produce valid results.
Skipping the test ensures that it runs only on systems where its results are meaningful.
...
testcases/kernel/sched/cfs-scheduler/starvation.c static void setup(void) · line 114
Quoted 3 lines
testcases/kernel/sched/cfs-scheduler/starvation.c +3
-0+if (tst_has_slow_kconfig ()) +tst_brk (TCONF ,"Skip test due to slow kernel configuration" ); +Reviewed-by Petr Vorel <[email protected]>
Quoted 3 lines
I suppose you have strong reason for this (e.g. it's failing on some slow machine). I'm testing it on emulated riscv64 to see whether it's needed to be skipped. Because due
CONFIG_LATENCYTOPit will always be skipped on Tumbleweed.
OK, I was able to reproduce TFAIL on emulated riscv64, which runs ~ 4m 30s, with timeout increased to 0h 05m 46s:
tst_tmpdir.c:316: TINFO: Using /tmp/LTP_sta3UIB6E as tmpdir (tmpfs filesystem)
tst_test.c:1893: TINFO: LTP version: 20240930
tst_test.c:1897: TINFO: Tested kernel: 6.12.6-1-default #1 SMP PREEMPT_DYNAMIC Thu Dec 19 17:23:25 UTC 2024 (fb072de) riscv64 tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz'
tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution tst_test.c:1728: TINFO: Timeout per run is 0h 02m 00s
starvation.c:98: TINFO: Setting affinity to CPU 0
starvation.c:52: TINFO: CPU did 120000000 loops in 226566us
tst_test.c:1736: TINFO: Updating max runtime to 0h 03m 46s
tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz'
tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution tst_test.c:1728: TINFO: Timeout per run is 0h 05m 46s
starvation.c:148: TFAIL: Scheduller starvation reproduced.
But it works on emulated x86_64.
tst_tmpdir.c:316: TINFO: Using /tmp/LTP_staIt0g73 as tmpdir (tmpfs filesystem)
tst_test.c:1893: TINFO: LTP version: 20240930-148-g4539bfbc7
tst_test.c:1897: TINFO: Tested kernel: 6.13.0-rc4-1.g4a5c6a6-default #1 SMP PREEMPT_DYNAMIC Sun Dec 22 22:11:35 UTC 2024 (4a5c6a6) x86_64 tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz'
tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution
tst_test.c:1728: TINFO: Timeout per run is 0h 02m 00s
starvation.c:98: TINFO: Setting affinity to CPU 0
starvation.c:52: TINFO: CPU did 120000000 loops in 34189us
tst_test.c:1736: TINFO: Updating max runtime to 0h 00m 34s
tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz'
tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution
tst_test.c:1728: TINFO: Timeout per run is 0h 02m 34s
starvation.c:150: TPASS: Haven't reproduced scheduller starvation.
I wonder if we can quit with TCONF only on certain runtime increase. We would also need to take -l (number of loops into an account).
Kind regards,
Petr
Quoted 2 lines
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 2/2] starvation: skip test on slow kernel
Quoted 64 lines · Petr Vorel
Quoted 11 lines
Hi Li,
Quoted 6 lines
Systems with slow kernel configurations may not meet the performance requirements necessary for the starvation test to produce valid results.
Skipping the test ensures that it runs only on systems where its results are meaningful.
...
Quoted 2 lines
(quoted diff) +2
-0+ if (tst_has_slow_kconfig()) + tst_brk(TCONF, "Skip test due to slow kernelconfiguration");
Quoted 6 lines
Quoted 1 line
+
Reviewed-by Petr Vorel <[email protected]>I suppose you have strong reason for this (e.g. it's failing on some slow machine). I'm testing it on emulated riscv64 to see whether it's needed
to be
Quoted 1 line
skipped. Because due
CONFIG_LATENCYTOPit will always be skipped onTumbleweed.
OK, I was able to reproduce TFAIL on emulated riscv64, which runs ~ 4m 30s, with
timeout increased to 0h 05m 46s:tst_tmpdir.c:316: TINFO: Using
/tmp/LTP_sta3UIB6Eas tmpdir (tmpfs filesystem)
tst_test.c:1893: TINFO: LTP version: 20240930 tst_test.c:1897: TINFO: Tested kernel: 6.12.6-1-default #1 SMPPREEMPT_DYNAMICThu Dec 19 17:23:25 UTC 2024 (fb072de) riscv64 tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO:CONFIG_LATENCYTOPkernel option detected which might slow the execution
tst_test.c:1728: TINFO: Timeout per run is 0h 02m 00s starvation.c:98: TINFO: Setting affinity to CPU 0 starvation.c:52: TINFO: CPU did 120000000 loops in 226566us tst_test.c:1736: TINFO: Updating max runtime to 0h 03m 46s tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO:CONFIG_LATENCYTOPkernel option detected which might slow the execution
tst_test.c:1728: TINFO: Timeout per run is 0h 05m 46s starvation.c:148: TFAIL: Scheduller starvation reproduced.But it works on emulated
x86_64.tst_tmpdir.c:316: TINFO: Using
/tmp/LTP_staIt0g73as tmpdir (tmpfs filesystem)
tst_test.c:1893: TINFO: LTP version: 20240930-148-g4539bfbc7 tst_test.c:1897: TINFO: Tested kernel: 6.13.0-rc4-1.g4a5c6a6-default #1 SMPPREEMPT_DYNAMICSun Dec 22 22:11:35 UTC 2024 (4a5c6a6)x86_64tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO:CONFIG_LATENCYTOPkernel option detected which might slow the execution
tst_test.c:1728: TINFO: Timeout per run is 0h 02m 00s starvation.c:98: TINFO: Setting affinity to CPU 0 starvation.c:52: TINFO: CPU did 120000000 loops in 34189us tst_test.c:1736: TINFO: Updating max runtime to 0h 00m 34s tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO:CONFIG_LATENCYTOPkernel option detected which might slow the execution
tst_test.c:1728: TINFO: Timeout per run is 0h 02m 34s starvation.c:150: TPASS: Haven't reproduced scheduller starvation.
Quoted 4 lines
I wonder if we can quit with TCONF only on certain runtime increase. We would
also need to take -l (number of loops into an account).
The simplest way is to extend 'max_runtime' when detecting slow configs for only starvation.c test.
Just multiply the argument of tst_set_max_runtime(timeout) for starvation.c.
I didn't do that because Philip (our Scheduler expert) told me that not much sense to run this on debug-kernel.
(I don't know that Tumbleweed enables CONFIG_LATENCYTOP by default)
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Li, all,
[ Cc others to get broather feedback ]
Quoted 7 lines
This refines the handling of timeouts for tests running on systems with slow kernel configurations (kconfigs).
Previously, the
max_runtimewas multiplied globally when slow kconfigs were detected, which inadvertently prolonged the runtime of all tests usingmax_runtimefor control.
Quoted 5 lines
This patch corrects that behavior by applying the multiplication specifically to timeouts, ensuring it only affects the intended operations without impacting other tests.
Fixes 2da30df24 ("lib: multiply the max_runtime if detect slow kconfigs")
Thanks for handling this, I overlooked it on 27th, thus review it now.
Multiplying whole timeout instead of max_runtime helps to hide longer timeout from the test which uses detection via tst_remaining_runtime(). I.e. previously it behaved on slow config as LTP_RUNTIME_MUL=4, now as LTP_TIMEOUT_MUL=4. Good idea. IMHO good enough (Martin previously suggested [1] to add a new tst_test flag to identify tests which exit when runtime expires).
Some measurements on my Tumbleweed VM, which is detected as slow due CONFIG_LATENCYTOP:
TEST | 2da30df24~ | 2da30df24 | this patch --------------------------------------------------|------------------------------------- swapping01.c (calls tst_remaining_runtime()) | 0h 10m 30s | 0h 40m 30s | 0h 12m 00s tst_fuzzy_sync01.c (calls tst_remaining_runtime())| 0h 03m 00s | 0h 10m 30s | 0h 04m 30s tst_cgroup02.c (default timeout 0h 00m 30s) | 0h 00m 30s | 0h 00m 30s | 0h 02m 00s test_runtime01.c (.max_runtime = 4, calls | 0h 00m 34s | 0h 00m 46s | 0h 02m 04s
tst_remaining_runtime())starvation.c (calls tst_remaining_runtime() only | 0h 01m 05s | 0h 02m 50s | 0h 02m 34s
to detect failure) |
=> Tests which call tst_remaining_runtime() runs slightly longer, but IMHO that's OK. Other tests (regardless if with the default runtime or these which set .max_runtime) run 4* longer as expected.
Kind regards,
Petr
[1] https://lore.kernel.org/ltp/[email protected]/
Quoted 28 lines
Signed-off-by Li Wang <[email protected]>--- lib/tst_test.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)lib/tst_test.c +3
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned int timeout)if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ; +return timeout * timeout_mul; }
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 18 lines · Petr Vorel
Hi Li, all,
[ Cc others to get broather feedback ]
Quoted 7 lines
This refines the handling of timeouts for tests running on systems with slow kernel configurations (kconfigs).
Previously, the
max_runtimewas multiplied globally when slow kconfigs were detected, which inadvertently prolonged the runtime of all tests usingmax_runtimefor control.Quoted 5 lines
This patch corrects that behavior by applying the multiplication specifically to timeouts, ensuring it only affects the intended operations without impacting other tests.
Fixes 2da30df24 ("lib: multiply the max_runtime if detect slowkconfigs")
Quoted 8 lines
Thanks for handling this, I overlooked it on 27th, thus review it now.
Multiplying whole timeout instead of
max_runtimehelps to hide longer timeout
from the test which uses detection viatst_remaining_runtime(). I.e. previously
it behaved on slow config as LTP_RUNTIME_MUL=4, now as LTP_TIMEOUT_MUL=4.
Yes, the benefit of multiplying TIMEOUT (on a slow system) is not only to avoid increasing the actual execution time of the test, but also to give
the system more time to wait for the test to complete the final work.
Original:
| -- timeout -- | -- max_runtime -- |
Previous:
| -- timeout -- | -------- max_runtime * 4 -------- |
Now:
| -------- timeout * 4 -------- | -- max_runtime -- |
Quoted 3 lines
Good idea. IMHO good enough (Martin previously suggested [1] to add a new
tst_testflag to identify tests which exit when runtime expires).
Introduce a new tst_test flag to split the max_runtime into two parts can resolve it as well, but the disadvantage might make people hard to understand the LTP time controlling, if go with timeout, max_runtime, max_exetime
I think 'simple+uselful' is beautiful unless we need to complex it in the future.
Quoted 23 lines
Reviewed-by Petr Vorel <[email protected]>Some measurements on my Tumbleweed VM, which is detected as slow due
CONFIG_LATENCYTOP:TEST | 2da30df24~ |
2da30df24| this patch--------------------------------------------------|-------------------------------------
swapping01.c(callstst_remaining_runtime()) | 0h 10m 30s | 0h 40m
30s | 0h 12m 00stst_fuzzy_sync01.c(calls tst_remaining_runtime())| 0h 03m 00s | 0h 10m 30s | 0h 04m 30stst_cgroup02.c(default timeout 0h 00m 30s) | 0h 00m 30s | 0h 00m
30s | 0h 02m 00stest_runtime01.c(.max_runtime = 4, calls | 0h 00m 34s | 0h 00m
46s | 0h 02m 04stst_remaining_runtime())starvation.c(callstst_remaining_runtime()only | 0h 01m 05s | 0h 02m 50s | 0h 02m 34s
to detect failure) |
Quoted 7 lines
=> Tests which call
tst_remaining_runtime()runs slightly longer, but IMHO that's OK. Other tests (regardless if with the default runtime or these which
set .max_runtime) run 4* longer as expected.Tested-by Petr Vorel <[email protected]>
The longer time is not because call tst_remaining_runtime(), it just comes from
the ' timeout *= 4' while detecting slow configs, as you can see, the original default
timeout is 30s, and with multiple 4 it become 120s (which is +2mins), all the test with
this patch shows that 2 more mins there.
But, that does not mean the test executing time is really adding 2 minutes, it just
means having that timeout value. We need to use time ./swapping01 to evaluate the real test time, and I didn't find any more delay with this method :).
Thanks for the comparison, actually I did some tests for RHEL and got a good result.
Quoted 37 lines
Kind regards,
Petr[1]
https://lore.kernel.org/ltp/[email protected]/lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int
Quoted 20 lines
Signed-off-by Li Wang <[email protected]>--- lib/tst_test.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)lib/tst_test.c +0
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned inttimeout)
Quoted 8 lines
lib/tst_test.c +3
-0if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ; +return timeout * timeout_mul; }
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 33 lines · Petr Vorel
Quoted 19 lines
Hi Li, all,
[ Cc others to get broather feedback ]
Quoted 7 lines
This refines the handling of timeouts for tests running on systems with slow kernel configurations (kconfigs).
Previously, the
max_runtimewas multiplied globally when slow kconfigs were detected, which inadvertently prolonged the runtime of all tests usingmax_runtimefor control.Quoted 5 lines
This patch corrects that behavior by applying the multiplication specifically to timeouts, ensuring it only affects the intended operations without impacting other tests.
Fixes 2da30df24 ("lib: multiply the max_runtime if detect slowkconfigs")
Quoted 7 lines
Thanks for handling this, I overlooked it on 27th, thus review it now.
Multiplying whole timeout instead of
max_runtimehelps to hide longer timeout
from the test which uses detection viatst_remaining_runtime(). I.e. previously
it behaved on slow config as LTP_RUNTIME_MUL=4, now as LTP_TIMEOUT_MUL=4.Yes, the benefit of multiplying TIMEOUT (on a slow system) is not only to avoid increasing the actual execution time of the test, but also to give
the system more time to wait for the test to complete the final work.
Quoted 8 lines
Original:
| -- timeout -- | --max_runtime-- |Previous:
| -- timeout -- | --------max_runtime* 4 -------- |Now:
| -------- timeout * 4 -------- | --max_runtime-- |
Later it'd be nice to document this simple timeline (also with LTP_RUNTIME_MUL and LTP_TIMEOUT_MUL) in sphinx docs (/** */). Or, it could be in lib/README.md, but I would like to convert also this page to sphinx.
Quoted 10 lines
Quoted 2 lines
Good idea. IMHO good enough (Martin previously suggested [1] to add a new
tst_testflag to identify tests which exit when runtime expires).Introduce a new
tst_testflag to split themax_runtimeinto two parts can resolve it as well, but the disadvantage might make people hard to understand the LTP time controlling, if go with timeout,max_runtime,max_exetime
I think 'simple+uselful' is beautiful unless we need to complex it in the future.
+1, I fully agree.
Quoted 38 lines
Quoted 22 lines
Reviewed-by Petr Vorel <[email protected]>Some measurements on my Tumbleweed VM, which is detected as slow due
CONFIG_LATENCYTOP:TEST | 2da30df24~ |
2da30df24| this patch--------------------------------------------------|-------------------------------------
swapping01.c(callstst_remaining_runtime()) | 0h 10m 30s | 0h 40m
30s | 0h 12m 00stst_fuzzy_sync01.c(calls tst_remaining_runtime())| 0h 03m 00s | 0h 10m 30s | 0h 04m 30stst_cgroup02.c(default timeout 0h 00m 30s) | 0h 00m 30s | 0h 00m
30s | 0h 02m 00stest_runtime01.c(.max_runtime = 4, calls | 0h 00m 34s | 0h 00m
46s | 0h 02m 04stst_remaining_runtime())starvation.c(callstst_remaining_runtime()only | 0h 01m 05s | 0h 02m 50s | 0h 02m 34s
to detect failure) |Quoted 6 lines
=> Tests which call
tst_remaining_runtime()runs slightly longer, but IMHO that's OK. Other tests (regardless if with the default runtime or these which
set .max_runtime) run 4* longer as expected.Tested-by Petr Vorel <[email protected]>The longer time is not because call
tst_remaining_runtime(), it just comes from
the ' timeout *= 4' while detecting slow configs, as you can see, the original default
timeout is 30s, and with multiple 4 it become 120s (which is +2mins), all the test with
this patch shows that 2 more mins there.
Quoted 5 lines
But, that does not mean the test executing time is really adding 2 minutes, it just
means having that timeout value. We need to usetime ./swapping01to evaluate the real test time, and I didn't find any more delay with this method :).
Yes, I noticed that (measuring just test_runtime01.c, where it's nicely visible).
Quoted 2 lines
Thanks for the comparison, actually I did some tests for RHEL and got a good result.
I'm OK with whole result. I'd be happier if we could avoid TCONF of starvation, but let's discuss this on that patch.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Li, all,
thinking about it twice, I vote for merging the patchset.
It's an improvement to the current state. And we can solve starvation later.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 6 lines · Petr Vorel
Hi Li, all,
thinking about it twice, I vote for merging the patchset. It's an improvement to the current state. And we can solve starvation later.
Yes, the patch only extends the timeout value and does not impact the real execution time of LTP.
starvation.c is a dedicated test we can treat separately (if people still want to run it on slow kernel).
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
In reply to Li WangHi!
Quoted 4 lines
Yes, the benefit of multiplying TIMEOUT (on a slow system) is not only to avoid increasing the actual execution time of the test, but also to give
the system more time to wait for the test to complete the final work.
Quoted 8 lines
Original:
| -- timeout -- | --max_runtime-- |Previous:
| -- timeout -- | --------max_runtime* 4 -------- |Now:
| -------- timeout * 4 -------- | --max_runtime-- |
The problems I see here:
There are tests, I think this covers mostly the IO stress tests, where the max runtime may be significantly larger than the timeout, so multiplying only the timeout may not be enough there.
I wanted to eventually move to a shorter default timeout, e.g. 10s once we have enough max_runtime anotation in the testcases.
So in the end we may eventually need both max_runtime and runtime in the tst_test structure.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 12 lines · Cyril Hrubis
Hi!
Quoted 2 lines
Yes, the benefit of multiplying TIMEOUT (on a slow system) is not only to avoid increasing the actual execution time of the test, but also to
give
Quoted 8 lines
the system more time to wait for the test to complete the final work.
Original:
| -- timeout -- | --max_runtime-- |Previous:
| -- timeout -- | --------max_runtime* 4 -------- |
Quoted 9 lines
Quoted 2 lines
Now:
| -------- timeout * 4 -------- | --max_runtime-- |The problems I see here:
There are tests, I think this covers mostly the IO stress tests, where the max runtime may be significantly larger than the timeout, so multiplying only the timeout may not be enough there.
I did a quick grep that some ltp-aiodio tests set it to 1800 sec, which only 8/91 occupation in the LTP testcases/, I'm not sure if it's worth adding a new field for those few stress tests.
And with the previous method, the multiple 4 max_runtime for 1800s is 2hours per test up limit, I can't imagine how long we will get eventually in the whole test time.
Maybe another way is to create a separate function in a header like aio_common.h (or in high-level dir) for handling that significantly larger runtime tests.
BTW, we have TST_UNLIMITED_RUNTIME choice or, invoke tst_set_max_runtime() in somehow.
Quoted 3 lines
I wanted to eventually move to a shorter default timeout, e.g. 10s once we have enough
max_runtimeanotation in the testcases.
That's fine, but most tests do not have .max_runtime. I guess what we should consider now is making them have enough timeout time on the slow kernel, isn't it?
$ git grep .max_runtime | wc -l 91
$ git grep .max_runtime cve/cve-2014-0196.c: .max_runtime = 60, cve/cve-2015-3290.c: .max_runtime = 180, cve/cve-2016-7117.c: .max_runtime = 60, cve/cve-2017-2671.c: .max_runtime = 40, kernel/controllers/cgroup/cgroup_core03.c: .max_runtime = 20, kernel/crypto/af_alg02.c: .max_runtime = 20, kernel/crypto/af_alg07.c: .max_runtime = 150, kernel/crypto/pcrypt_aead01.c: .max_runtime = 300, kernel/fs/fs_fill/fs_fill.c: .max_runtime = 300, kernel/fs/fsplough/fsplough.c: tst_set_max_runtime(bufsize * loop_count / (8 * 1024 * 1024)); kernel/fs/fsx-linux/fsx-linux.c: .max_runtime = 1800, kernel/fs/read_all/read_all.c: tst_res(TINFO, "Worker timeout set to 10%% of max_runtime: %dms", kernel/fs/read_all/read_all.c: .max_runtime = 100, kernel/io/ltp-aiodio/aio-stress.c: .max_runtime = 1800, kernel/io/ltp-aiodio/aiocp.c: .max_runtime = 1800, kernel/io/ltp-aiodio/aiodio_append.c: .max_runtime = 1800, kernel/io/ltp-aiodio/aiodio_sparse.c: .max_runtime = 1800, kernel/io/ltp-aiodio/dio_append.c: .max_runtime = 1800, kernel/io/ltp-aiodio/dio_read.c: .max_runtime = 1800, kernel/io/ltp-aiodio/dio_sparse.c: .max_runtime = 1800, kernel/io/ltp-aiodio/dio_truncate.c: .max_runtime = 1800, kernel/mem/ksm/ksm02.c: tst_set_max_runtime(32 * (size / DEFAULT_MEMSIZE)); kernel/mem/ksm/ksm02.c: .max_runtime = 32, kernel/mem/ksm/ksm04.c: tst_set_max_runtime(32 * (size / DEFAULT_MEMSIZE)); kernel/mem/ksm/ksm04.c: .max_runtime = 32, kernel/mem/mmapstress/mmapstress01.c: .max_runtime = 12, kernel/mem/mtest01/mtest01.c: .max_runtime = 300, kernel/mem/mtest06/mmap1.c: .max_runtime = 180, kernel/mem/mtest06/mmap3.c: .max_runtime = 60, kernel/mem/mtest07/mallocstress.c: .max_runtime = 600, kernel/mem/oom/oom01.c: .max_runtime = TST_UNLIMITED_RUNTIME, kernel/mem/oom/oom02.c: .max_runtime = TST_UNLIMITED_RUNTIME, kernel/mem/oom/oom03.c: .max_runtime = TST_UNLIMITED_RUNTIME, kernel/mem/oom/oom04.c: .max_runtime = TST_UNLIMITED_RUNTIME, kernel/mem/oom/oom05.c: .max_runtime = TST_UNLIMITED_RUNTIME, kernel/mem/swapping/swapping01.c: .max_runtime = 600, kernel/mem/thp/thp04.c: .max_runtime = 150, kernel/mem/tunable/min_free_kbytes.c: .max_runtime = TST_UNLIMITED_RUNTIME, kernel/pty/pty03.c: .max_runtime = 30, kernel/pty/pty05.c: .max_runtime = 150, kernel/pty/pty06.c: .max_runtime = 150, kernel/pty/pty07.c: .max_runtime = 150, kernel/sched/cfs-scheduler/cfs_bandwidth01.c: .max_runtime = 20, kernel/sched/cfs-scheduler/starvation.c: tst_set_max_runtime(timeout); kernel/security/dirtyc0w_shmem/dirtyc0w_shmem.c: .max_runtime = 120, kernel/security/kallsyms/kallsyms.c: .max_runtime = 60, kernel/sound/snd_seq01.c: .max_runtime = 60, kernel/sound/snd_timer01.c: .max_runtime = 150, kernel/syscalls/bind/bind06.c: .max_runtime = 300, kernel/syscalls/copy_file_range/copy_file_range01.c: .max_runtime = 5 kernel/syscalls/fallocate/fallocate06.c: .max_runtime = 120, kernel/syscalls/fork/fork13.c: .max_runtime = 600, kernel/syscalls/fsync/fsync02.c: .max_runtime = 300, kernel/syscalls/gettimeofday/gettimeofday02.c: .max_runtime = 10, kernel/syscalls/inotify/inotify06.c: .max_runtime = 600, kernel/syscalls/inotify/inotify09.c: .max_runtime = 150, kernel/syscalls/inotify/inotify12.c: .max_runtime = 10, kernel/syscalls/ioctl/ioctl_sg01.c: .max_runtime = 3600, kernel/syscalls/ipc/msgstress/msgstress01.c: .max_runtime = 180, kernel/syscalls/ipc/semget/semget05.c: tst_set_max_runtime(maxsems / 200); kernel/syscalls/ipc/shmctl/shmctl05.c: .max_runtime = 10, kernel/syscalls/keyctl/keyctl02.c: .max_runtime = 60, kernel/syscalls/landlock/landlock04.c: .max_runtime = 360, kernel/syscalls/madvise/madvise06.c: .max_runtime = 60, kernel/syscalls/madvise/madvise11.c: .max_runtime = 30, kernel/syscalls/migrate_pages/migrate_pages03.c: .max_runtime = 300, kernel/syscalls/move_pages/move_pages12.c: .max_runtime = 240, kernel/syscalls/nice/nice05.c: .max_runtime = 3, kernel/syscalls/perf_event_open/perf_event_open02.c: .max_runtime = 72 kernel/syscalls/perf_event_open/perf_event_open03.c: .max_runtime = 300, kernel/syscalls/preadv2/preadv203.c: .max_runtime = 60, kernel/syscalls/readahead/readahead02.c: * speed, sometime test timeout when the default max_runtime is used up. kernel/syscalls/readahead/readahead02.c: tst_set_max_runtime(test.max_runtime + (usec + usec_ra) / 1000000); kernel/syscalls/readahead/readahead02.c: tst_set_max_runtime(1 + testfile_size / (DEFAULT_FILESIZE/32)); kernel/syscalls/readahead/readahead02.c: .max_runtime = 30, kernel/syscalls/request_key/request_key03.c: .max_runtime = 20, kernel/syscalls/sendfile/sendfile09.c: .max_runtime = 120, kernel/syscalls/sendmsg/sendmsg03.c: .max_runtime = 150, kernel/syscalls/set_mempolicy/set_mempolicy01.c: tst_set_max_runtime(test.max_runtime * (1 << nodes->cnt/16)); kernel/syscalls/set_mempolicy/set_mempolicy01.c: .max_runtime = 600, kernel/syscalls/setsockopt/setsockopt06.c: .max_runtime = 270, kernel/syscalls/setsockopt/setsockopt07.c: .max_runtime = 150, kernel/syscalls/swapoff/swapoff01.c: .max_runtime = 60, kernel/syscalls/swapon/swapon01.c: .max_runtime = 60, kernel/syscalls/timerfd/timerfd_settime02.c: .max_runtime = 150, kernel/syscalls/writev/writev03.c: .max_runtime = 75, network/can/cve/can_bcm01.c: .max_runtime = 30, network/netstress/netstress.c: .max_runtime = 300, network/nfs/nfs_stress/nfs05_make_tree.c: .max_runtime = 300, network/packet/fanout01.c: .max_runtime = 180, network/sockets/vsock01.c: .max_runtime = 60,
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Hi all,
[ Cc Andrea due aiodio ]
Quoted 24 lines · Cyril Hrubis
Quoted 13 lines
Hi!
Quoted 2 lines
Yes, the benefit of multiplying TIMEOUT (on a slow system) is not only to avoid increasing the actual execution time of the test, but also to
give
Quoted 7 lines
the system more time to wait for the test to complete the final work.
Original:
| -- timeout -- | --max_runtime-- |Previous:
| -- timeout -- | --------max_runtime* 4 -------- |Quoted 8 lines
Quoted 2 lines
Now:
| -------- timeout * 4 -------- | --max_runtime-- |The problems I see here:
There are tests, I think this covers mostly the IO stress tests, where the max runtime may be significantly larger than the timeout, so multiplying only the timeout may not be enough there.
Quoted 7 lines
I did a quick grep that some ltp-aiodio tests set it to 1800 sec, which only 8/91 occupation in the LTP testcases/, I'm not sure if it's worth adding a new field for those few stress tests.
And with the previous method, the multiple 4
max_runtimefor 1800s is 2hours per test up limit, I can't imagine how long we will get eventually in the whole test time.
Agree. This solution is quite simple. I would also hesitate to add new field for few tests.
It's questionable whether .max_runtime = 1800 is reasonable for aiodio tests at all (see below). I'll try to have look and send a patch to shorten it. For some tests it'd be nice to have some calculation instead of blindly expecting they run 30 min (because we have many runs of these tests and vast majority runs slowly).
OTOH it's questionable how much time we should spent with these tests, see Jan Kara [1]:
So the performance of this test is irrelevant because combining buffered reads with direct IO writes was always in "better don't do it" territory. Definitely not if you care about performance.
NOTE: Jan's comment is only for aiodio_sparse.c and dio_sparse.c, but I suppose this is related to all LTP aiodio tests (at least dio_append.c does also buffered reads with O_DIRECT, dio_read.c combines buffered writes with O_DIRECT).
[1] https://lore.kernel.org/ltp/20240722090012.mlvkaenuxar2x3vr@quack3/
Quoted 21 lines
Maybe another way is to create a separate function in a header like
aio_common.h(or in high-level dir) for handling that significantly larger runtime tests.BTW, we have
TST_UNLIMITED_RUNTIMEchoice or, invoketst_set_max_runtime()in somehow.Quoted 2 lines
I wanted to eventually move to a shorter default timeout, e.g. 10s once we have enough
max_runtimeanotation in the testcases.That's fine, but most tests do not have .max_runtime. I guess what we should consider now is making them have enough timeout time on the slow kernel, isn't it?
$ git grep .max_runtime | wc -l 91
Quoted 13 lines
$ git grep .max_runtime cve/cve-2014-0196.c: .max_runtime = 60, cve/cve-2015-3290.c: .max_runtime = 180, cve/cve-2016-7117.c: .max_runtime = 60, cve/cve-2017-2671.c: .max_runtime = 40, kernel/controllers/cgroup/cgroup_core03.c: .max_runtime = 20, kernel/crypto/af_alg02.c: .max_runtime = 20, kernel/crypto/af_alg07.c: .max_runtime = 150, kernel/crypto/pcrypt_aead01.c: .max_runtime = 300, kernel/fs/fs_fill/fs_fill.c: .max_runtime = 300, kernel/fs/fsplough/fsplough.c: tst_set_max_runtime(bufsize * loop_count / (8 * 1024 * 1024)); kernel/fs/fsx-linux/fsx-linux.c: .max_runtime = 1800,
Just looking into some aiodio tests.
We have 22 variants of these tests (none without params). The longest is fsx22 (-N 100000) due much higher -N than other tests and the default (1000). The highest runtime I see on this test is the current mainline (6.12): 1m 31s.
Quoted 6 lines
kernel/fs/read_all/read_all.c: tst_res(TINFO, "Worker timeout set
to 10%% ofmax_runtime: %dms",kernel/fs/read_all/read_all.c: .max_runtime = 100,kernel/io/ltp-aiodio/aio-stress.c: .max_runtime = 1800,kernel/io/ltp-aiodio/aiocp.c: .max_runtime = 1800,kernel/io/ltp-aiodio/aiodio_append.c: .max_runtime = 1800,
For example aiodio_append runs on 1 sec on Tumbleweed.
Quoted 1 line
kernel/io/ltp-aiodio/aiodio_sparse.c: .max_runtime = 1800,
This is a valid one, it runs without parameters 9m 39s (ADSP000).
But we have several parameters (it runs 38x with a different parameters) Most of them run below 7s.
These are running longer, obviously combination of high -w and -n:
ADSP037 44s (-w 18192k -n 512)
ADSP038 1m 44s (-w 18192k -n 1000)
We simplified our life with blindly saying everything runs 30 min, but vast majority don't.
Also this is a long run on the current mainline kernel (slow down in 6.6 [1]). It runs on older kernels ~1 or 2 min.
Quoted 1 line
kernel/io/ltp-aiodio/dio_append.c: .max_runtime = 1800,
This runs ~ 32 sec on Tumbleweed.
Quoted 2 lines
kernel/io/ltp-aiodio/dio_read.c: .max_runtime = 1800,kernel/io/ltp-aiodio/dio_sparse.c: .max_runtime = 1800,
The only long running is ADSP039 (dio_sparse without params): 9m 37s
Quoted 30 lines
kernel/io/ltp-aiodio/dio_truncate.c: .max_runtime = 1800,kernel/mem/ksm/ksm02.c: tst_set_max_runtime(32 * (size /DEFAULT_MEMSIZE));kernel/mem/ksm/ksm02.c: .max_runtime = 32,kernel/mem/ksm/ksm04.c: tst_set_max_runtime(32 * (size /DEFAULT_MEMSIZE));kernel/mem/ksm/ksm04.c: .max_runtime = 32,kernel/mem/mmapstress/mmapstress01.c: .max_runtime = 12,kernel/mem/mtest01/mtest01.c: .max_runtime = 300, kernel/mem/mtest06/mmap1.c: .max_runtime = 180, kernel/mem/mtest06/mmap3.c: .max_runtime = 60,
kernel/mem/mtest07/mallocstress.c: .max_runtime = 600,kernel/mem/oom/oom01.c: .max_runtime =TST_UNLIMITED_RUNTIME,kernel/mem/oom/oom02.c: .max_runtime =TST_UNLIMITED_RUNTIME,kernel/mem/oom/oom03.c: .max_runtime =TST_UNLIMITED_RUNTIME,kernel/mem/oom/oom04.c: .max_runtime =TST_UNLIMITED_RUNTIME,kernel/mem/oom/oom05.c: .max_runtime =TST_UNLIMITED_RUNTIME,kernel/mem/swapping/swapping01.c: .max_runtime = 600,kernel/mem/thp/thp04.c: .max_runtime = 150,kernel/mem/tunable/min_free_kbytes.c: .max_runtime =TST_UNLIMITED_RUNTIME,kernel/pty/pty03.c: .max_runtime = 30, kernel/pty/pty05.c: .max_runtime = 150, kernel/pty/pty06.c: .max_runtime = 150, kernel/pty/pty07.c: .max_runtime = 150,
kernel/sched/cfs-scheduler/cfs_bandwidth01.c: .max_runtime = 20,kernel/sched/cfs-scheduler/starvation.c:
tst_set_max_runtime(timeout);kernel/security/dirtyc0w_shmem/dirtyc0w_shmem.c: .max_runtime = 120,kernel/security/kallsyms/kallsyms.c: .max_runtime = 60,
Quoted 12 lines
kernel/sound/snd_seq01.c: .max_runtime = 60, kernel/sound/snd_timer01.c: .max_runtime = 150, kernel/syscalls/bind/bind06.c: .max_runtime = 300, kernel/syscalls/copy_file_range/copy_file_range01.c: .max_runtime = 5 kernel/syscalls/fallocate/fallocate06.c: .max_runtime = 120, kernel/syscalls/fork/fork13.c: .max_runtime = 600, kernel/syscalls/fsync/fsync02.c: .max_runtime = 300, kernel/syscalls/gettimeofday/gettimeofday02.c: .max_runtime = 10, kernel/syscalls/inotify/inotify06.c: .max_runtime = 600, kernel/syscalls/inotify/inotify09.c: .max_runtime = 150, kernel/syscalls/inotify/inotify12.c: .max_runtime = 10, kernel/syscalls/ioctl/ioctl_sg01.c: .max_runtime = 3600,
FYI Martin set 1 hour due long setup on systems with big RAM. Real runtime is 1 sec on 8GB RAM.
Kind regards,
Petr
Quoted 37 lines
kernel/syscalls/ipc/msgstress/msgstress01.c: .max_runtime = 180,kernel/syscalls/ipc/semget/semget05.c: tst_set_max_runtime(maxsems / 200); kernel/syscalls/ipc/shmctl/shmctl05.c: .max_runtime = 10, kernel/syscalls/keyctl/keyctl02.c: .max_runtime = 60, kernel/syscalls/landlock/landlock04.c: .max_runtime = 360, kernel/syscalls/madvise/madvise06.c: .max_runtime = 60, kernel/syscalls/madvise/madvise11.c: .max_runtime = 30,
kernel/syscalls/migrate_pages/migrate_pages03.c: .max_runtime = 300,kernel/syscalls/move_pages/move_pages12.c: .max_runtime = 240,kernel/syscalls/nice/nice05.c: .max_runtime = 3,kernel/syscalls/perf_event_open/perf_event_open02.c: .max_runtime = 72kernel/syscalls/perf_event_open/perf_event_open03.c: .max_runtime = 300,kernel/syscalls/preadv2/preadv203.c: .max_runtime = 60,kernel/syscalls/readahead/readahead02.c: * speed, sometime test
timeout when the defaultmax_runtimeis used up.kernel/syscalls/readahead/readahead02.c:
tst_set_max_runtime(test.max_runtime + (usec + usec_ra) / 1000000);kernel/syscalls/readahead/readahead02.c:
tst_set_max_runtime(1 + testfile_size / (DEFAULT_FILESIZE/32));kernel/syscalls/readahead/readahead02.c: .max_runtime = 30,kernel/syscalls/request_key/request_key03.c: .max_runtime = 20,kernel/syscalls/sendfile/sendfile09.c: .max_runtime = 120,kernel/syscalls/sendmsg/sendmsg03.c: .max_runtime = 150,kernel/syscalls/set_mempolicy/set_mempolicy01.c:
tst_set_max_runtime(test.max_runtime * (1 << nodes->cnt/16));kernel/syscalls/set_mempolicy/set_mempolicy01.c: .max_runtime = 600,kernel/syscalls/setsockopt/setsockopt06.c: .max_runtime = 270,kernel/syscalls/setsockopt/setsockopt07.c: .max_runtime = 150,kernel/syscalls/swapoff/swapoff01.c: .max_runtime = 60,kernel/syscalls/swapon/swapon01.c: .max_runtime = 60,kernel/syscalls/timerfd/timerfd_settime02.c: .max_runtime = 150,kernel/syscalls/writev/writev03.c: .max_runtime = 75,network/can/cve/can_bcm01.c: .max_runtime = 30, network/netstress/netstress.c: .max_runtime = 300, network/nfs/nfs_stress/nfs05_make_tree.c: .max_runtime = 300, network/packet/fanout01.c: .max_runtime = 180, network/sockets/vsock01.c: .max_runtime = 60,
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
In reply to Li WangHi!
Quoted 8 lines
I did a quick grep that some ltp-aiodio tests set it to 1800 sec, which only 8/91 occupation in the LTP testcases/, I'm not sure if it's worth adding a new field for those few stress tests.
And with the previous method, the multiple 4
max_runtimefor 1800s is 2hours per test up limit, I can't imagine how long we will get eventually in the whole test time.
Quoted 6 lines
Maybe another way is to create a separate function in a header like
aio_common.h(or in high-level dir) for handling that significantly larger runtime tests.BTW, we have
TST_UNLIMITED_RUNTIMEchoice or, invoketst_set_max_runtime()in somehow.
I stil think that misusing max_runtime, which is supposed to be upper bound for the actual test runtime was a mistake.
Maybe we should have called the max_runtime a timeout and add runtime for tests that needs it. That way we would have timeout compromising of two parts, one would be the 30s that is used for all tests and second part from the tst_test structure. And then the sum of these two would be multiplied by the timeout multipliers. Then we would have a runtime, which would be used only by tests that call tst_remaining_runtime().
The overall test timeout would be then:
(default_30s_timeout + tst_test->timeout) * TIMEOUT_MUL + tst_test->runtime * RUNTIME_MUL
What do you think?
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 13 lines
Hi!
Quoted 11 lines
I did a quick grep that some ltp-aiodio tests set it to 1800 sec, which only 8/91 occupation in the LTP testcases/, I'm not sure if it's worth adding a new field for those few stress tests.
And with the previous method, the multiple 4
max_runtimefor 1800s is 2hours per test up limit, I can't imagine how long we will get eventually in the whole test time.Maybe another way is to create a separate function in a header like
aio_common.h(or in high-level dir) for handling that significantly larger runtime tests.
Quoted 5 lines
Quoted 2 lines
BTW, we have
TST_UNLIMITED_RUNTIMEchoice or, invoketst_set_max_runtime()in somehow.I stil think that misusing
max_runtime, which is supposed to be upper bound for the actual test runtime was a mistake.
Do you want to separate timeout for setup() and for actual test run? Which one would be prolonged in case of "slow" kernels? (e.g. this patch).
Or you want to fix -iN?
Also for docs purposes it might be useful to list long running tests. Fortunately there are just few tests which calls tst_set_max_runtime() for dynamically set timeout.
Quoted 7 lines
Maybe we should have called the
max_runtimea timeout and add runtime for tests that needs it. That way we would have timeout compromising of two parts, one would be the 30s that is used for all tests and second part from thetst_teststructure. And then the sum of these two would be multiplied by the timeout multipliers. Then we would have a runtime, which would be used only by tests that calltst_remaining_runtime().
Quoted 5 lines
The overall test timeout would be then:
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+tst_test->runtime*RUNTIME_MULWhat do you think?
Timeout is for setup function, right? e.g. for ioctl_sg01? If yes, timeout is too generic, IMHO many people will think that it's a general test timeout. I would think about general name.
The above formula should be written in the docs in the separate description and this section should be linked in TIMEOUT_MUL and RUNTIME_MUL description [1].
Maybe part of library README.md [2], which itself should be also moved to sphinx docs.
Kind regards,
Petr
[1] https://linux-test-project.readthedocs.io/en/latest/users/setup_tests.html#library-environment-variables
[2] https://github.com/linux-test-project/ltp/blob/master/lib/README.md
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Hi!
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
I did a quick grep that some ltp-aiodio tests set it to 1800 sec, which only 8/91 occupation in the LTP testcases/, I'm not sure if it's worth adding a new field for those few stress tests.
And with the previous method, the multiple 4
max_runtimefor 1800s is 2hours per test up limit, I can't imagine how long we will get eventually in the whole test time.Maybe another way is to create a separate function in a header like
aio_common.h(or in high-level dir) for handling that significantly larger runtime tests.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
BTW, we have
TST_UNLIMITED_RUNTIMEchoice or, invoketst_set_max_runtime()in somehow.I stil think that misusing
max_runtime, which is supposed to be upper bound for the actual test runtime was a mistake.Do you want to separate timeout for
setup()and for actual test run? Which one would be prolonged in case of "slow" kernels? (e.g. this patch).
Currently the test timeout applies both for the setup, single test iteration and cleanup. So the short answer is both.
Quoted 1 line
Or you want to fix -iN?
After each test iteration the child signals the parent that the iteration has been finished and that restarts the timeout timer.
The test overall timeout is computed in set_timeout() and currently is computed as:
results->timeout = tst_multiply_timeout(timeout) + results->max_runtime;
And the tst_multiply_tiemout() would then be changed into:
return (DEFAULT_TIMEOUT + tst_test->timeout) * timeout_mul;
Quoted 3 lines
Also for docs purposes it might be useful to list long running tests. Fortunately there are just few tests which calls
tst_set_max_runtime()for dynamically set timeout.
That would be actually easier, because we could do:
- add special value TST_RUNTIME_TIMEOUT
- allow test to change timeout only if TST_RUNTIME_TIMEOUT was set in tst_test
Then all long running test would have either tst_test->timeout or tst_test->runtime set.
Technically we would need two special timeout values TST_UNLIMITED_TIMEOUT and TST_RUNTIME_TIMEOUT.
Quoted 9 lines
Quoted 8 lines
Maybe we should have called the
max_runtimea timeout and add runtime for tests that needs it. That way we would have timeout compromising of two parts, one would be the 30s that is used for all tests and second part from thetst_teststructure. And then the sum of these two would be multiplied by the timeout multipliers. Then we would have a runtime, which would be used only by tests that calltst_remaining_runtime().The overall test timeout would be then:
Quoted 7 lines
Quoted 3 lines
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+tst_test->runtime*RUNTIME_MULWhat do you think?
Timeout is for setup function, right? e.g. for
ioctl_sg01? If yes, timeout is too generic, IMHO many people will think that it's a general test timeout. I would think about general name.
Not only, the default 30s timeout is for the whole testrun for tests that are quick enough. We have a lot of tests that run just for less than 1s even on small embedded boards.
Quoted 2 lines
The above formula should be written in the docs in the separate description and this section should be linked in
TIMEOUT_MULandRUNTIME_MULdescription [1].
Yes.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 22 lines
Hi!
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
I did a quick grep that some ltp-aiodio tests set it to 1800 sec, which only 8/91 occupation in the LTP testcases/, I'm not sure if it's worth adding a new field for those few stress tests.
And with the previous method, the multiple 4
max_runtimefor 1800s is 2hours per test up limit, I can't imagine how long we will get eventually in the whole test time.Maybe another way is to create a separate function in a header like
aio_common.h(or in high-level dir) for handling that significantly larger runtime tests.Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
BTW, we have
TST_UNLIMITED_RUNTIMEchoice or, invoketst_set_max_runtime()in somehow.I stil think that misusing
max_runtime, which is supposed to be upper bound for the actual test runtime was a mistake.Do you want to separate timeout for
setup()and for actual test run? Which one would be prolonged in case of "slow" kernels? (e.g. this patch).
Quoted 7 lines
Currently the test timeout applies both for the setup, single test iteration and cleanup. So the short answer is both.
Quoted 1 line
Or you want to fix -iN?
After each test iteration the child signals the parent that the iteration has been finished and that restarts the timeout timer.
OK, -iN is already solved. Obviously, otherwise testing -i2000 would not work.
Quoted 14 lines
The test overall timeout is computed in
set_timeout()and currently is computed as:results->timeout = tst_multiply_timeout(timeout) + results->max_runtime;And the
tst_multiply_tiemout()would then be changed into:return (DEFAULT_TIMEOUT + tst_test->timeout) * timeout_mul;Quoted 3 lines
Also for docs purposes it might be useful to list long running tests. Fortunately there are just few tests which calls
tst_set_max_runtime()for dynamically set timeout.
Quoted 5 lines
That would be actually easier, because we could do:
- add special value
TST_RUNTIME_TIMEOUT
- allow test to change timeout only ifTST_RUNTIME_TIMEOUTwas set intst_test
And if value does not get changed it's the default value. Also name is a bit confusing (you suggest to have members "timeout", "runtime"), which suggest the special value is related to both.
OK, flag which would allow us to see that time will be changed.
Quoted 5 lines
Then all long running test would have either
tst_test->timeoutortst_test->runtimeset.Technically we would need two special timeout values
TST_UNLIMITED_TIMEOUTandTST_RUNTIME_TIMEOUT.
OK, TST_UNLIMITED_TIMEOUT is equivalent of:
#define TST_UNLIMITED_RUNTIME (-1)
Maybe have just single definition TST_UNLIMITED, which could be used in both tst_test->timeout and tst_test->runtime? But that's just an implementation detail.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Maybe we should have called the
max_runtimea timeout and add runtime for tests that needs it. That way we would have timeout compromising of two parts, one would be the 30s that is used for all tests and second part from thetst_teststructure. And then the sum of these two would be multiplied by the timeout multipliers. Then we would have a runtime, which would be used only by tests that calltst_remaining_runtime().
OK, the point of whole change is to separate some general timeout (30 sec) from runtime (used for tst_remaining_runtime()), right?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The overall test timeout would be then:
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+tst_test->runtime*RUNTIME_MUL
(default_30s_timeout + tst_test->timeout) * TIMEOUT_MUL is meant for setup or cleanup and library overhead, tst_test->runtime * RUNTIME_MUL for running test function?
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
What do you think?
Timeout is for setup function, right? e.g. for
ioctl_sg01? If yes, timeout is too generic, IMHO many people will think that it's a general test timeout. I would think about general name.Not only, the default 30s timeout is for the whole testrun for tests that are quick enough. We have a lot of tests that run just for less than 1s even on small embedded boards.
And yes, with properly set data, 30s could be carefully lowered down a bit.
Kind regards,
Petr
Quoted 4 lines
Quoted 2 lines
The above formula should be written in the docs in the separate description and this section should be linked in
TIMEOUT_MULandRUNTIME_MULdescription [1].Yes.
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Hi!
Quoted 12 lines
And if value does not get changed it's the default value. Also name is a bit confusing (you suggest to have members "timeout", "runtime"), which suggest the special value is related to both.
OK, flag which would allow us to see that time will be changed.
Quoted 5 lines
Then all long running test would have either
tst_test->timeoutortst_test->runtimeset.Technically we would need two special timeout values
TST_UNLIMITED_TIMEOUTandTST_RUNTIME_TIMEOUT.
Quoted 7 lines
OK,
TST_UNLIMITED_TIMEOUTis equivalent of:#define
TST_UNLIMITED_RUNTIME(-1)Maybe have just single definition
TST_UNLIMITED, which could be used in bothtst_test->timeoutandtst_test->runtime? But that's just an implementation detail.
The UNLIMITED_RUNTIME would not be needed anymore. Because runtime will mean _only_ for how long will a few tests spend in the main loop. Infinite loop does not make any sense. The tst_runtime will be basically renamed to timeout and TST_UNLIMITED_RUNTIME becomes TST_UNLIMITED_TIMEOUT.
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Maybe we should have called the
max_runtimea timeout and add runtime for tests that needs it. That way we would have timeout compromising of two parts, one would be the 30s that is used for all tests and second part from thetst_teststructure. And then the sum of these two would be multiplied by the timeout multipliers. Then we would have a runtime, which would be used only by tests that calltst_remaining_runtime().OK, the point of whole change is to separate some general timeout (30 sec) from runtime (used for
tst_remaining_runtime()), right?
The point is to separate timeout, which is a guess on upper bound on the time the test will spend executing from a runtime which is the exact time a few tests will spend looping in the test function.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The overall test timeout would be then:
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+tst_test->runtime*RUNTIME_MUL(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MULis meant for setup or cleanup and library overhead,tst_test->runtime*RUNTIME_MULfor running test function?
No, it's the other way around. The timeout is a guessed upper bound for a test execution time. It's for everything the test does and in most of the cases only the default timeout (which is kind of safety measure) is sufficient. Then there are tests that do some work that is not time bound, e.g. I/O. In that case we set the timeout in the tst_test structure to some value we measured in practice and that plus the default timeout will become the overall test timeout.
Then we have a few testcases that do a loop in the test function that takes exact time. In that case we know that we spend exactly runtime in the test function, but there is a setup and cleanup as well. So we add the default timeout to the runtime to get the overall timeout. There may be also a case where the test setup for such test takes some time, in that case we would set both the timeout and runtime in the tst_test structure. The timeout would be upper bound for the test setup and runtime would be exactly for how long will the test function loop.
Mainly this would make sure that if timeout part of the overall test time limit gets multiplied, because we are running on a slow system, the runtime will not. Because we could control the timeout and runtime separately.
Quoted 5 lines
Quoted 3 lines
Not only, the default 30s timeout is for the whole testrun for tests that are quick enough. We have a lot of tests that run just for less than 1s even on small embedded boards.
And yes, with properly set data, 30s could be carefully lowered down a bit.
That was my long term plan.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 71 lines · Cyril Hrubis
Hi!
Quoted 1 line
And if value does not get changed it's the default value. Also name is a
bit
Quoted 1 line
confusing (you suggest to have members "timeout", "runtime"), which
suggest the
Quoted 7 lines
special value is related to both.
OK, flag which would allow us to see that time will be changed.
Quoted 2 lines
Then all long running test would have either
tst_test->timeoutortst_test->runtimeset.Quoted 8 lines
Quoted 2 lines
Technically we would need two special timeout values
TST_UNLIMITED_TIMEOUTandTST_RUNTIME_TIMEOUT.OK,
TST_UNLIMITED_TIMEOUTis equivalent of:#define
TST_UNLIMITED_RUNTIME(-1)Maybe have just single definition
TST_UNLIMITED, which could be used inboth
Quoted 1 line
tst_test->timeoutandtst_test->runtime? But that's just animplementation
Quoted 1 line
detail.
The
UNLIMITED_RUNTIMEwould not be needed anymore. Because runtime will mean _only_ for how long will a few tests spend in the main loop. Infinite loop does not make any sense. Thetst_runtimewill be basically renamed to timeout andTST_UNLIMITED_RUNTIMEbecomesTST_UNLIMITED_TIMEOUT.Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Maybe we should have called the
max_runtimea timeout and addruntime
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
for tests that needs it. That way we would have timeout
compromising of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
two parts, one would be the 30s that is used for all tests and
second
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
part from the
tst_teststructure. And then the sum of these twowould be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiplied by the timeout multipliers. Then we would have a
runtime,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
which would be used only by tests that call
tst_remaining_runtime().Quoted 2 lines
OK, the point of whole change is to separate some general timeout (30
sec) from
Quoted 1 line
runtime (used for
tst_remaining_runtime()), right?The point is to separate timeout, which is a guess on upper bound on the time the test will spend executing from a runtime which is the exact time a few tests will spend looping in the test function.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The overall test timeout would be then:
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+
tst_test->runtime*RUNTIME_MULQuoted 2 lines
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MULis meant forsetup or
Quoted 1 line
cleanup and library overhead,
tst_test->runtime*RUNTIME_MULforrunning test
Quoted 1 line
function?
No, it's the other way around. The timeout is a guessed upper bound for a test execution time. It's for everything the test does and in most of the cases only the default timeout (which is kind of safety measure) is sufficient. Then there are tests that do some work that is not time bound, e.g. I/O. In that case we set the timeout in the
tst_teststructure to some value we measured in practice and that plus the default timeout will become the overall test timeout.
Quoted 9 lines
Then we have a few testcases that do a loop in the test function that takes exact time. In that case we know that we spend exactly runtime in the test function, but there is a setup and cleanup as well. So we add the default timeout to the runtime to get the overall timeout. There may be also a case where the test setup for such test takes some time, in that case we would set both the timeout and runtime in the
tst_teststructure. The timeout would be upper bound for the test setup and runtime would be exactly for how long will the test function loop.
Fair enough, I agree with that.
Thus we will have tst_test->timeout used for setup time control, and tst_test->runtime means the exact executable time of the main function, tst_remaining_runtime() only rely on tst_test->runtime to count.
As you pointed out above:
(default_30s_timeout + tst_test->timeout) * TIMEOUT_MUL +tst_test->runtime * RUNTIME_MUL
But, questions come back to the item, which part should be extended when detecting slow kernel configs? the TIMEOUT_MUL?
If so it looks only renaming something based on my patch no essential changes.
Quoted 20 lines
Mainly this would make sure that if timeout part of the overall test time limit gets multiplied, because we are running on a slow system, the runtime will not. Because we could control the timeout and runtime separately.
Quoted 5 lines
Quoted 3 lines
Not only, the default 30s timeout is for the whole testrun for tests that are quick enough. We have a lot of tests that run just for less than 1s even on small embedded boards.
And yes, with properly set data, 30s could be carefully lowered down a
bit.
That was my long term plan.
--
Cyril Hrubis
[email protected]
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 91 lines · Li Wang
Quoted 71 lines · Cyril Hrubis
Hi!
Quoted 1 line
And if value does not get changed it's the default value. Also name is
a bit
Quoted 1 line
confusing (you suggest to have members "timeout", "runtime"), which
suggest the
Quoted 7 lines
special value is related to both.
OK, flag which would allow us to see that time will be changed.
Quoted 2 lines
Then all long running test would have either
tst_test->timeoutortst_test->runtimeset.Quoted 8 lines
Quoted 2 lines
Technically we would need two special timeout values
TST_UNLIMITED_TIMEOUTandTST_RUNTIME_TIMEOUT.OK,
TST_UNLIMITED_TIMEOUTis equivalent of:#define
TST_UNLIMITED_RUNTIME(-1)Maybe have just single definition
TST_UNLIMITED, which could be used inboth
Quoted 1 line
tst_test->timeoutandtst_test->runtime? But that's just animplementation
Quoted 1 line
detail.
The
UNLIMITED_RUNTIMEwould not be needed anymore. Because runtime will mean _only_ for how long will a few tests spend in the main loop. Infinite loop does not make any sense. Thetst_runtimewill be basically renamed to timeout andTST_UNLIMITED_RUNTIMEbecomesTST_UNLIMITED_TIMEOUT.Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Maybe we should have called the
max_runtimea timeout and addruntime
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
for tests that needs it. That way we would have timeout
compromising of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
two parts, one would be the 30s that is used for all tests and
second
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
part from the
tst_teststructure. And then the sum of these twowould be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiplied by the timeout multipliers. Then we would have a
runtime,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
which would be used only by tests that call
tst_remaining_runtime().Quoted 2 lines
OK, the point of whole change is to separate some general timeout (30
sec) from
Quoted 1 line
runtime (used for
tst_remaining_runtime()), right?The point is to separate timeout, which is a guess on upper bound on the time the test will spend executing from a runtime which is the exact time a few tests will spend looping in the test function.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The overall test timeout would be then:
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+
tst_test->runtime*RUNTIME_MULQuoted 2 lines
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MULis meant forsetup or
Quoted 1 line
cleanup and library overhead,
tst_test->runtime*RUNTIME_MULforrunning test
Quoted 1 line
function?
No, it's the other way around. The timeout is a guessed upper bound for a test execution time. It's for everything the test does and in most of the cases only the default timeout (which is kind of safety measure) is sufficient. Then there are tests that do some work that is not time bound, e.g. I/O. In that case we set the timeout in the
tst_teststructure to some value we measured in practice and that plus the default timeout will become the overall test timeout.Quoted 9 lines
Then we have a few testcases that do a loop in the test function that takes exact time. In that case we know that we spend exactly runtime in the test function, but there is a setup and cleanup as well. So we add the default timeout to the runtime to get the overall timeout. There may be also a case where the test setup for such test takes some time, in that case we would set both the timeout and runtime in the
tst_teststructure. The timeout would be upper bound for the test setup and runtime would be exactly for how long will the test function loop.Fair enough, I agree with that.
Thus we will have
tst_test->timeoutused for setup time control, andtst_test->runtimemeans the exact executable time of the main function,tst_remaining_runtime()only rely ontst_test->runtimeto count.
Quoted 19 lines
As you pointed out above:
(default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+tst_test->runtime*RUNTIME_MULBut, questions come back to the item, which part should be extended when detecting slow kernel configs? the
TIMEOUT_MUL?
If so it looks only renaming something based on my patch no essential changes.Quoted 6 lines
Mainly this would make sure that if timeout part of the overall test time limit gets multiplied, because we are running on a slow system, the runtime will not. Because we could control the timeout and runtime separately.
Ah, you mean we multiply the overall test time limit results->timeout, right?
e.g.
results->timeout = (default_30s_timeout + tst_test->timeout) *
TIMEOUT_MUL + tst_test->runtime * RUNTIME_MUL;
if (tst_has_slow_kconfig())
results->timeout *= 4;Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Not only, the default 30s timeout is for the whole testrun for tests that are quick enough. We have a lot of tests that run just for less than 1s even on small embedded boards.
Quoted 10 lines
Quoted 1 line
And yes, with properly set data, 30s could be carefully lowered down a
bit.
That was my long term plan.
--
Cyril Hrubis
[email protected]--
Regards,
Li Wang
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Hi!
Quoted 8 lines
Ah, you mean we multiply the overall test time limit
results->timeout, right?
e.g.results->timeout = (default_30s_timeout + tst_test->timeout) * TIMEOUT_MUL + tst_test->runtime * RUNTIME_MUL; if (tst_has_slow_kconfig()) results->timeout *= 4;
That would work too, but since the runtime will be always constant (once only the test that call tst_remaning_runtime() use runtime and the rest of the tests is moved to timeout) we may as well multiply the timeout part.
However this detail does not matter that much, the most imporatant part is the clear separation of the guessed upper bound and the actual runtime that is used to controll how long should the loop in the test spin.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 15 lines
Hi!
Quoted 8 lines
Ah, you mean we multiply the overall test time limit
results->timeout, right?
e.g.results->timeout = (default_30s_timeout + tst_test->timeout) * TIMEOUT_MUL + tst_test->runtime * RUNTIME_MUL; if (tst_has_slow_kconfig()) results->timeout *= 4;That would work too, but since the runtime will be always constant (once only the test that call
tst_remaning_runtime()use runtime and the rest of the tests is moved to timeout) we may as well multiply the timeout part.
Quoted 4 lines
However this detail does not matter that much, the most imporatant part is the clear separation of the guessed upper bound and the actual runtime that is used to controll how long should the loop in the test spin.
Thanks this for explanation.
BTW do you plan to send a patch soon? If not I would prefer to temporarily revert 2da30df24e (or just *= 4), which causes some tests to timeout.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 20 lines · Petr Vorel
Quoted 15 lines
Hi!
Quoted 8 lines
Ah, you mean we multiply the overall test time limit
results->timeout, right?
e.g.results->timeout = (default_30s_timeout + tst_test->timeout) * TIMEOUT_MUL + tst_test->runtime * RUNTIME_MUL; if (tst_has_slow_kconfig()) results->timeout *= 4;That would work too, but since the runtime will be always constant (once only the test that call
tst_remaning_runtime()use runtime and the rest of the tests is moved to timeout) we may as well multiply the timeout part.Quoted 4 lines
However this detail does not matter that much, the most imporatant part is the clear separation of the guessed upper bound and the actual runtime that is used to controll how long should the loop in the test spin.
Quoted 5 lines
Thanks this for explanation.
BTW do you plan to send a patch soon? If not I would prefer to temporarily revert
2da30df24e(or just *= 4), which causes some tests to timeout.
Petr, Cyril, all,
I am now working on the new patch, but it's not guaranteed the new one will be ready for the coming release.
It's fine to apply this one, at least it addresses the side effects and extend
timeout time for most tests.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Hi All,
I boldly merge this patch because it could temporarily resolve the side effects and help extend the time (on slow kernel) for most tests.
And, based on the above discussion I will fully work on the new timeout+runtime method improvement this week.
Hope we can get that down for the coming release.
Thanks everyone for the suggestions!
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
In reply to Cyril HrubisQuoted 10 lines · Cyril Hrubis
I stil think that misusing
max_runtime, which is supposed to be upper bound for the actual test runtime was a mistake.Maybe we should have called the
max_runtimea timeout and add runtime for tests that needs it. That way we would have timeout compromising of two parts, one would be the 30s that is used for all tests and second part from thetst_teststructure. And then the sum of these two would be multiplied by the timeout multipliers. Then we would have a runtime, which would be used only by tests that calltst_remaining_runtime().
Quoted 5 lines
The overall test timeout would be then:
(
default_30s_timeout+tst_test->timeout) *TIMEOUT_MUL+tst_test->runtime*RUNTIME_MULWhat do you think?
Hi,
sorry but I still don't follow the logic in the math above. I agree that "runtime" should control test iteration and "timeout" should be a hard limit for test execution. But then it doesn't make sense to add these two numbers and RUNTIME_MUL would be pointless. Instead, the total timeout (for single testcase/filesystem test) should be calculated like this:
default_30s_timeout * TIMEOUT_MUL + MAX(MAX(1, tst_test->timeout) * TIMEOUT_MUL, tst_test->runtime)
If you want to force a different runtime value, it should be done through the -I command line parameter. We could also replace the "duration" logic in testrun() with tst_remaining_runtime() which will allow looping tests for fixed amount of time by default just by setting the tst_test->runtime attribute, without any loop code inside the test function itself.
Signature · 10 lines
Martin Doucha [email protected] SW Quality Engineer SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Hi!
Quoted 6 lines
If you want to force a different runtime value, it should be done through the -I command line parameter. We could also replace the "duration" logic in
testrun()withtst_remaining_runtime()which will allow looping tests for fixed amount of time by default just by setting thetst_test->runtimeattribute, without any loop code inside the test function itself.
The problem we have is that we currently use max_runtime for two different purposes. See my other emails where I explain how would I like to decouple these two different things that are currently bundled together in max_runtime.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD
Quoted 4 lines · Cyril Hrubis
The problem we have is that we currently use
max_runtimefor two different purposes. See my other emails where I explain how would I like to decouple these two different things that are currently bundled together inmax_runtime.
I got that. I'm just pointing out that your current proposal still keeps them somewhat coupled. The only relationship between runtime and timeout should be that tst_test->runtime will automatically set a lower bound for tst_test->timeout * TIMEOUT_MUL. Nothing more.
Signature · 10 lines
Martin Doucha [email protected] SW Quality Engineer SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp
[LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
In reply to Li WangHi Li, Cyril, all,
...
Quoted 12 lines
+++
b/lib/tst_test.clib/tst_test.c +0
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int timeout) · line 1707
Quoted 5 lines
lib/tst_test.c +2
-0@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned int timeout)if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ;
FYI this change, merged as 893ca0abe7 ("lib: multiply the timeout if detect slow kconfigs") caused a regression on *all* tests which use tst_net.sh.
First, it detects slow config on everything which uses struct tst_test, which are unfortunately some tools at testcases/lib/:
$ git grep -l "struct tst_test" testcases/lib/*.c testcases/lib/tst_device.c // not obvious reason, might be removed testcases/lib/tst_get_free_pids.c // force messages to be printed from new library testcases/lib/tst_ns_create.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_ns_exec.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_run_shell.c // not obvious reason, might be removed
Besides unimportant fact that slow config detection is an unnecessary slowdown on these tools, the problem is that it prints messages to stderr, which causes breakage.
Previously tst_ns_exec.c run just command passed by -c parameter:
$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" 0
Now it prints TINFO:
$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution 0
tst_rhost_run() in tst_net.sh runs binary on remote host:
output=$($rcmd "$sh_cmd" $out 2>&1 || echo 'RTERR')
Redirect stderr to stdout (2>&1) is likely needed for some tests. But tst_rhost_run() output is often parsed including therefore certain tools in testcases/lib/ have to print only expected output:
init_ltp_netspace()
{
...
pid="$(ROD tst_ns_create net,mnt)"
...
export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"I will probably solve it by adding yet another parameter to tst_rhost_run(), which ignores stderr and use it for tst_ns_{create,exec}. But maybe there is another solution (or another problem).
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Quoted 72 lines · Petr Vorel
Hi Li, Cyril, all,
...
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int
Quoted 12 lines
+++
b/lib/tst_test.clib/tst_test.c +0
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned inttimeout)
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int timeout) · line 1707
Quoted 5 lines
lib/tst_test.c +2
-0if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ;FYI this change, merged as
893ca0abe7("lib: multiply the timeout if detect slow
kconfigs") caused a regression on *all* tests which usetst_net.sh.First, it detects slow config on everything which uses struct
tst_test, which are unfortunately some tools at testcases/lib/:$ git grep -l "struct tst_test" testcases/lib/*.c testcases/lib/tst_device.c // not obvious reason, might be removed testcases/lib/tst_get_free_pids.c // force messages to be printed from new library testcases/lib/tst_ns_create.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_ns_exec.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_run_shell.c // not obvious reason, might be removedBesides unimportant fact that slow config detection is an unnecessary slowdown
on these tools, the problem is that it prints messages to stderr, which causes
breakage.Previously
tst_ns_exec.crun just command passed by -c parameter:$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" 0Now it prints TINFO:
$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution 0
tst_rhost_run()intst_net.shruns binary on remote host:output=$($rcmd "$sh_cmd" $out 2>&1 || echo 'RTERR')Redirect stderr to stdout (2>&1) is likely needed for some tests. But
tst_rhost_run()output is often parsed including therefore certain tools in testcases/lib/ have to print only expected output:
init_ltp_netspace(){ ... pid="$(ROD tst_ns_create net,mnt)" ... export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"
Quoted 6 lines
I will probably solve it by adding yet another parameter to
tst_rhost_run(),
which ignores stderr and use it for tst_ns_{create,exec}. But maybe there is
another solution (or another problem).
Or, another way is just to set 'tst_test->timeout == TST_UNLIMITED_TIMEOUT' in those testcase/lib tools.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Quoted 81 lines · Petr Vorel
Quoted 73 lines
Hi Li, Cyril, all,
...
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int
Quoted 12 lines
+++
b/lib/tst_test.clib/tst_test.c +0
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned inttimeout)
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int timeout) · line 1707
Quoted 5 lines
lib/tst_test.c +2
-0if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ;FYI this change, merged as
893ca0abe7("lib: multiply the timeout if detect slow
kconfigs") caused a regression on *all* tests which usetst_net.sh.First, it detects slow config on everything which uses struct
tst_test, which are unfortunately some tools at testcases/lib/:$ git grep -l "struct tst_test" testcases/lib/*.c testcases/lib/tst_device.c // not obvious reason, might be removed testcases/lib/tst_get_free_pids.c // force messages to be printed from new library testcases/lib/tst_ns_create.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_ns_exec.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_run_shell.c // not obvious reason, might be removedBesides unimportant fact that slow config detection is an unnecessary slowdown
on these tools, the problem is that it prints messages to stderr, which causes
breakage.Previously
tst_ns_exec.crun just command passed by -c parameter:$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" 0Now it prints TINFO:
$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution 0
tst_rhost_run()intst_net.shruns binary on remote host:output=$($rcmd "$sh_cmd" $out 2>&1 || echo 'RTERR')Redirect stderr to stdout (2>&1) is likely needed for some tests. But
tst_rhost_run()output is often parsed including therefore certain tools in testcases/lib/ have to print only expected output:
init_ltp_netspace(){ ... pid="$(ROD tst_ns_create net,mnt)" ... export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"Quoted 5 lines
I will probably solve it by adding yet another parameter to
tst_rhost_run(),
which ignores stderr and use it for tst_ns_{create,exec}. But maybe there is
another solution (or another problem).
Quoted 2 lines
Or, another way is just to set '
tst_test->timeout==TST_UNLIMITED_TIMEOUT' in those testcase/lib tools.
This would be an easiest fix, but OTOH it's another hack which depends on the implementation (I would prefer to get rid of the need to use struct tst_test for these tools instead).
Anyway, I'll send a patch to fix tst_net.sh, but ideas how to improve and unify testcases/lib are welcome.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
In reply to Li Wang...
Quoted 14 lines
Quoted 7 lines
init_ltp_netspace(){ ... pid="$(ROD tst_ns_create net,mnt)" ... export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"Quoted 5 lines
I will probably solve it by adding yet another parameter to
tst_rhost_run(),
which ignores stderr and use it for tst_ns_{create,exec}. But maybe there is
another solution (or another problem).
Quoted 2 lines
Or, another way is just to set '
tst_test->timeout==TST_UNLIMITED_TIMEOUT' in those testcase/lib tools.
Or, we could have yet another flag TST_SKIP_SLOW_DETECTION which could be used for these tools.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Quoted 16 lines · Petr Vorel
...
Quoted 10 lines
Quoted 10 lines
init_ltp_netspace(){ ... pid="$(ROD tst_ns_create net,mnt)" ... export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"I will probably solve it by adding yet another parameter to
tst_rhost_run(),
which ignores stderr and use it for tst_ns_{create,exec}. But maybethere
Quoted 2 lines
Quoted 2 lines
is
another solution (or another problem).
Quoted 8 lines
Quoted 1 line
Or, another way is just to set '
tst_test->timeout==
TST_UNLIMITED_TIMEOUT'Quoted 1 line
in those testcase/lib tools.
Or, we could have yet another flag
TST_SKIP_SLOW_DETECTIONwhich could be used
for these tools.
Sounds better. But maybe name it TST_NO_SLOW_CONFIG_CHECK ?
lib/tst_test.c +0 -0
@@ -1702,8 +1702,10 @@ unsigned int tst_multiply_timeout(unsigned int
timeout)
if (timeout < 1)
tst_brk(TBROK, "timeout must to be >= 1! (%d)", timeout);
+#ifndef TST_NO_SLOW_CONFIG_CHECK
if (tst_has_slow_kconfig())
timeout *= 4;+#endif
return timeout * timeout_mul;
}
testcases/lib/tst_ns_exec.c +1 -0
@@ -14,6 +14,7 @@*/ #define TST_NO_DEFAULT_MAIN +#define TST_NO_SLOW_CONFIG_CHECK #include <stdio.h> #include <sys/wait.h>
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Quoted 22 lines · Li Wang
Quoted 18 lines · Petr Vorel
...
Quoted 10 lines
Quoted 10 lines
init_ltp_netspace(){ ... pid="$(ROD tst_ns_create net,mnt)" ... export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"I will probably solve it by adding yet another parameter to
tst_rhost_run(),
which ignores stderr and use it for tst_ns_{create,exec}. But maybethere
Quoted 5 lines
Quoted 2 lines
is
another solution (or another problem).Or, another way is just to set '
tst_test->timeout==
TST_UNLIMITED_TIMEOUT'
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
in those testcase/lib tools.
Or, we could have yet another flag
TST_SKIP_SLOW_DETECTIONwhich could be used
for these tools.Sounds better. But maybe name it
TST_NO_SLOW_CONFIG_CHECK?
Ah, typo with missing 'K'.
s/TST_NO_SLOW_CONFIG_CHECK/TST_NO_SLOW_KCONFIG_CHECK
Quoted 1 line
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
In reply to Li WangQuoted 50 lines · Petr Vorel
Quoted 17 lines
...
Quoted 10 lines
Quoted 10 lines
init_ltp_netspace(){ ... pid="$(ROD tst_ns_create net,mnt)" ... export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"I will probably solve it by adding yet another parameter to
tst_rhost_run(),
which ignores stderr and use it for tst_ns_{create,exec}. But maybethere
Quoted 2 lines
Quoted 2 lines
is
another solution (or another problem).Quoted 7 lines
Quoted 1 line
Or, another way is just to set '
tst_test->timeout==
TST_UNLIMITED_TIMEOUT'Quoted 1 line
in those testcase/lib tools.
Or, we could have yet another flag
TST_SKIP_SLOW_DETECTIONwhich could be used
for these tools.Sounds better. But maybe name it
TST_NO_SLOW_CONFIG_CHECK?lib/tst_test.c +2
-0@@ -1702,8 +1702,10 @@ unsigned int tst_multiply_timeout(unsigned int timeout)if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +#ifndef TST_NO_SLOW_CONFIG_CHECK if (tst_has_slow_kconfig ()) timeout *=4 ; +#endif return timeout * timeout_mul;}
testcases/lib/tst_ns_exec.c +0
-0
testcases/lib/tst_ns_exec.c line 20
Quoted 7 lines
testcases/lib/tst_ns_exec.c +1
-0@@ -14,6 +14,7 @@*/ #define TST_NO_DEFAULT_MAIN +#define TST_NO_SLOW_CONFIG_CHECK #include <stdio.h> #include <sys/wait.h>
Looks useful, I vote to add it for all tools in testcases/lib/ which define TST_NO_DEFAULT_MAIN.
FYI I added other hardening, which prevents other potential problems with unwanted output from the library:
https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] Broken tests using tst_net.sh by 893ca0abe7 (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
In reply to Li WangQuoted 74 lines · Petr Vorel
Quoted 72 lines
Hi Li, Cyril, all,
...
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int
Quoted 12 lines
+++
b/lib/tst_test.clib/tst_test.c +0
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned inttimeout)
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int timeout) · line 1707
Quoted 5 lines
lib/tst_test.c +2
-0if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ;FYI this change, merged as
893ca0abe7("lib: multiply the timeout if detect slow
kconfigs") caused a regression on *all* tests which usetst_net.sh.First, it detects slow config on everything which uses struct
tst_test, which are unfortunately some tools at testcases/lib/:$ git grep -l "struct tst_test" testcases/lib/*.c testcases/lib/tst_device.c // not obvious reason, might be removed testcases/lib/tst_get_free_pids.c // force messages to be printed from new library testcases/lib/tst_ns_create.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_ns_exec.c // .forks_child = 1, Needed by SAFE_CLONE testcases/lib/tst_run_shell.c // not obvious reason, might be removedBesides unimportant fact that slow config detection is an unnecessary slowdown
on these tools, the problem is that it prints messages to stderr, which causes
breakage.Previously
tst_ns_exec.crun just command passed by -c parameter:$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" 0Now it prints TINFO:
$ ./tst_ns_exec 14536 net,mnt sh -c " cat /proc/sys/net/ipv6/conf/ltp_ns_veth1/disable_ipv6" tst_kconfig.c:88: TINFO: Parsing kernel config '/proc/config.gz' tst_kconfig.c:667: TINFO: CONFIG_LATENCYTOP kernel option detected which might slow the execution 0
tst_rhost_run()intst_net.shruns binary on remote host:output=$($rcmd "$sh_cmd" $out 2>&1 || echo 'RTERR')Redirect stderr to stdout (2>&1) is likely needed for some tests. But
tst_rhost_run()output is often parsed including therefore certain tools in testcases/lib/ have to print only expected output:
init_ltp_netspace(){ ... pid="$(ROD tst_ns_create net,mnt)" ... export LTP_NETNS="${LTP_NETNS:-tst_ns_exec $pid net,mnt}"
Quoted 3 lines
Quoted 3 lines
I will probably solve it by adding yet another parameter to
tst_rhost_run(),
which ignores stderr and use it for tst_ns_{create,exec}. But maybe there
Actually tst_ns_exec.c needs to be quiet to not clutter the output of the command it runs via tst_rhost_run() for netns (via $LTP_NETNS). Therefore adding an quiet option to tst_rhost_run() is not a solution.
Quoted 6 lines
Quoted 2 lines
is
another solution (or another problem).Or, another way is just to set '
tst_test->timeout==TST_UNLIMITED_TIMEOUT' in those testcase/lib tools.
Unfortunately this does not work. In the end I probably filter out 'TINFO:' messages in tst_rhost_run().
It'd be safer to avoid using library for tst_ns_exec.c, but it's also used in containers shell tests (netns_lib.sh, netns_sysfs.sh) and fs_bind tests (fs_bind_cloneNS*.sh).
Also, tst_ns_exec.c (and tst_ns_create.c) could be for netns replaced by 'ip netns' (revert back change in 2017 3fb501e04c ("net: use LTP ns_create/ns_exec")
Maybe other tools which use tst_ns_{create,exec,ifmove}.c could be replaced by some tools from util-linux (unshare ?). We consider having own binary as better than external dependency, but binary needs to be simple to avoid problems when LTP library changes.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] pidns05 timeout (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
In reply to Li WangQuoted 21 lines · Petr Vorel
Quoted 19 lines
Hi Li, Cyril, all,
...
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int
Quoted 12 lines
+++
b/lib/tst_test.clib/tst_test.c +0
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned inttimeout)
Quoted 2 lines
lib/tst_test.c +0
-0if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout);
Quoted 6 lines
Quoted 6 lines
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int timeout) · line 1707
Quoted 2 lines
lib/tst_test.c +2
-0+if (tst_has_slow_kconfig ()) + timeout *=4 ;FYI this change, merged as
893ca0abe7("lib: multiply the timeout if detect slow
kconfigs") caused a regression on *all* tests which usetst_net.sh.
...
FYI also at least pidns05.c sometimes fails due timeout with this change. On some of SLES product previously pidns05.c run for 3 sec. With this change it runs 12s and therefore timeouts.
In pidns05.c case child is run 5x. For each of this child we again detect if we run on slow config. Maybe we should have used struct tst_test member to cache the value.
What bothers me more that how much time we waste for whole LTP testing with repeatedly detecting slow config for all tests (runtest/syscalls has 1457 items, we run it more times for each product with different kernel cmdline parameters). I don't know what was supposed to be fixed by this feature, is it really worth of slowdown? Why not just set LTP_RUNTIME_MUL=2 on slow kernels? We could have tool which would 'exit 1' on "slow" kernel and 'exit 0' on normal kernel to do automatic detection, which could be run by frameworks just once.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] pidns05 timeout (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Quoted 24 lines
Quoted 23 lines · Petr Vorel
Quoted 22 lines
Hi Li, Cyril, all,
...
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int
Quoted 12 lines
+++
b/lib/tst_test.clib/tst_test.c +0
-3@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 ); -if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsigned inttimeout)
Quoted 5 lines
lib/tst_test.c +2
-0if (timeout <1 )tst_brk (TBROK ,"timeout must to be >= 1! (%d)" , timeout); +if (tst_has_slow_kconfig ()) + timeout *=4 ;
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
FYI this change, merged as
893ca0abe7("lib: multiply the timeout if detect slow
kconfigs") caused a regression on *all* tests which usetst_net.sh....
FYI also at least
pidns05.csometimes fails due timeout with this change. On some of SLES product previouslypidns05.crun for 3 sec. With this change it runs 12s and therefore timeouts.
I'm sorry for a wrong report. Looking about it twice there is "*** stack smashing detected ***: terminated" => some other problem, which causes slow down. IMHO it's not optimal to run the detection many times + basically now requiring kernel config for each LTP test, but performance impact is probably low.
Kind regards,
Petr
Quoted 14 lines
In
pidns05.ccase child is run 5x. For each of this child we again detect if we run on slow config. Maybe we should have used structtst_testmember to cache the value.What bothers me more that how much time we waste for whole LTP testing with repeatedly detecting slow config for all tests (runtest/syscalls has 1457 items, we run it more times for each product with different kernel cmdline parameters). I don't know what was supposed to be fixed by this feature, is it really worth of slowdown? Why not just set LTP_RUNTIME_MUL=2 on slow kernels? We could have tool which would 'exit 1' on "slow" kernel and 'exit 0' on normal kernel to do automatic detection, which could be run by frameworks just once.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] pidns05 timeout (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Quoted 34 lines · Petr Vorel
Quoted 10 lines
Quoted 9 lines · Petr Vorel
Quoted 8 lines
Hi Li, Cyril, all,
...
Quoted 4 lines
+++
b/lib/tst_test.clib/tst_test.c +0
-0@@ -555,9 +555,6 @@ static int multiply_runtime(int max_runtime)parse_mul (&runtime_mul,"LTP_RUNTIME_MUL" ,0.0099 ,100 );Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned
Quoted 7 lines
lib/tst_test.c +0
-3-if (tst_has_slow_kconfig ()) - max_runtime *=4 ; -return max_runtime * runtime_mul; }@@ -1706,6 +1703,9 @@ unsigned int tst_multiply_timeout(unsignedint
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
timeout)
Quoted 2 lines
(quoted diff) +0
-0if (timeout < 1) tst_brk(TBROK, "timeout must to be >= 1! (%d)",timeout);
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
lib/tst_test.c unsigned int tst_multiply_timeout(unsigned int timeout) · line 1707
Quoted 2 lines
lib/tst_test.c +2
-0+if (tst_has_slow_kconfig ()) + timeout *=4 ;FYI this change, merged as
893ca0abe7("lib: multiply the timeout if detect slow
kconfigs") caused a regression on *all* tests which usetst_net.sh....
FYI also at least
pidns05.csometimes fails due timeout with this change. On some of SLES product previouslypidns05.crun for 3 sec. With thischange it
Quoted 9 lines
Quoted 1 line
runs 12s and therefore timeouts.
I'm sorry for a wrong report. Looking about it twice there is "*** stack smashing detected ***: terminated" => some other problem, which causes slow down. IMHO it's not optimal to run the detection many times + basically now requiring kernel config for each LTP test, but performance impact is probably
low.
Yes, I understand your concern. I agree that we can avoid the kconfig detection for most faster test cases.
After reviewing my previous pre-release LTP test and compare to the current main branch test, it doesn't show much more time-consuming, maybe there is different HW/kernel factors, but the performance impact can be ignored.
Quoted 28 lines
Kind regards,
PetrQuoted 1 line
In
pidns05.ccase child is run 5x. For each of this child we againdetect if we
Quoted 1 line
run on slow config. Maybe we should have used struct
tst_testmember tocache
Quoted 3 lines
the value.
What bothers me more that how much time we waste for whole LTP testing
with
Quoted 1 line
repeatedly detecting slow config for all tests (runtest/syscalls has
1457 items,
Quoted 1 line
we run it more times for each product with different kernel cmdline
parameters).
Quoted 1 line
I don't know what was supposed to be fixed by this feature, is it really
worth
Quoted 1 line
of slowdown? Why not just set LTP_RUNTIME_MUL=2 on slow kernels? We
could have
Quoted 1 line
tool which would 'exit 1' on "slow" kernel and 'exit 0' on normal kernel
to do
Quoted 4 lines
automatic detection, which could be run by frameworks just once.
Kind regards,
Petr
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] pidns05 timeout (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Hi!
Quoted 7 lines
Yes, I understand your concern. I agree that we can avoid the kconfig detection for most faster test cases.
After reviewing my previous pre-release LTP test and compare to the current main branch test, it doesn't show much more time-consuming, maybe there is different HW/kernel factors, but the performance impact can be ignored.
I've just send a patch that caches the detection results, because in the case of pins05 we did parse the config for each SAFE_CLONE() call.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [REGRESSION] pidns05 timeout (was: [PATCH 1/2] lib: multiply the timeout if detect slow kconfigs)
Quoted 11 lines
Hi!
Quoted 7 lines
Yes, I understand your concern. I agree that we can avoid the kconfig detection for most faster test cases.
After reviewing my previous pre-release LTP test and compare to the current main branch test, it doesn't show much more time-consuming, maybe there is different HW/kernel factors, but the performance impact can be ignored.
I've just send a patch that caches the detection results, because in the case of pins05 we did parse the config for each
SAFE_CLONE()call.
Thanks! Elegant solution, I should have come up with static myself instead of complaining :).
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp