starvation: set a baseline for maximum runtime
RFC PATCH 1/1 starvation: set a baseline for maximum runtime
The commit ec14f4572 ("sched: starvation: Autocallibrate the timeout") introduced a runtime calibration mechanism to dynamically adjust test timeouts based on CPU speed.
While this works well for slower systems like microcontrollers or ARM boards, it struggles to determine appropriate runtimes for modern CPUs, especially when debugging kernels with significant overhead.
This patch introduces a baseline runtime (max_runtime = 600 seconds) to ensure the test does not timeout prematurely, even on modern CPUs or debug kernels. The calibrated runtime is compared against this baseline, and the greater value is used as the test timeout.
This change reduces the likelihood of timeouts while maintaining flexibility for slower systems.
Error log on debug-kernel:
...
starvation.c:98: TINFO: Setting affinity to CPU 0 starvation.c:52: TINFO: CPU did 120000000 loops in 52717us tst_test.c:1727: TINFO: Updating max runtime to 0h 00m 52s tst_test.c:1719: TINFO: Timeout per run is 0h 06m 16s starvation.c:148: TFAIL: Scheduller starvation reproduced.
...
From Philip Auld:
"The test sends a large number of signals as fast as possible. On the non-debug kernel both signal generation and signal deliver take 1usec in my traces (maybe actually less in real time but the timestamp has usec granularity).
But on the debug kernel these signal events take ~21usecs. A significant increase and given the large number of them this leads the starvation test to falsely report starvation when in fact it is just taking a lot longer.
In both debug and non-debug the kernel is doing the same thing. Both tasks are running as expected. It's just the timing is not working for the debug case.
Probably should waive this as expected failure on the debug variants."
--- testcases/kernel/sched/cfs-scheduler/starvation.c | 2 ++ 1 file changed, 2 insertions(+)
testcases/kernel/sched/cfs-scheduler/starvation.c +2 -0
@@ -108,6 +108,7 @@ static void setup(void)else timeout =callibrate () /1000 ; + timeout =MAX (timeout, test.max_runtime);tst_set_max_runtime (timeout); }
@@ -161,5 +162,6 @@ static struct tst_test test = { {"t:", &str_timeout, "Max timeout (default 240s)"}, {} }, + .max_runtime = 600, .needs_checkpoints = 1, };
Signature · 5 lines
2.47.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 7 lines
The commit
ec14f4572("sched: starvation: Autocallibrate the timeout") introduced a runtime calibration mechanism to dynamically adjust test timeouts based on CPU speed.While this works well for slower systems like microcontrollers or ARM boards, it struggles to determine appropriate runtimes for modern CPUs, especially when debugging kernels with significant overhead.
Wouldn't it be better to either skip the test on kernels with debuging confing options on? Or multiply the timeout we got from the callibration when we detect a debugging kernel?
The problem is that any number we put there will not be correct in a few years as CPU and RAM speed increase and the test will be effectively doing nothing because the default we put there will cover kernels that are overly slow on a future hardware.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 5 lines · Cyril Hrubis
Hi!
Quoted 4 lines
The commit
ec14f4572("sched: starvation: Autocallibrate the timeout") introduced a runtime calibration mechanism to dynamically adjust test timeouts based on CPU speed.
Quoted 8 lines
Quoted 3 lines
While this works well for slower systems like microcontrollers or ARM boards, it struggles to determine appropriate runtimes for modern CPUs, especially when debugging kernels with significant overhead.
Wouldn't it be better to either skip the test on kernels with debuging confing options on? Or multiply the timeout we got from the callibration when we detect a debugging kernel?
Well, we have not achieved a reliable way to detect debug kernels in LTP. While I looking at our RHEL9 kernel config file. The general kernel also enables things like "CONFIG_DEBUG_KERNEL=y".
# uname -r 5.14.0-533.el9.x86_64
# grep CONFIG_DEBUG_KERNEL /boot/config-5.14.0-533.el9.x86_64 CONFIG_DEBUG_KERNEL=y
Quoted 5 lines
The problem is that any number we put there will not be correct in a few years as CPU and RAM speed increase and the test will be effectively doing nothing because the default we put there will cover kernels that are overly slow on a future hardware.
Sounds reasonable. The hardcode baseline time is not a wise method, It is still possible not to satisfy some slower boards or new processors.
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 3 lines
Well, we have not achieved a reliable way to detect debug kernels in LTP. While I looking at our RHEL9 kernel config file. The general kernel also enables things like "
CONFIG_DEBUG_KERNEL=y".
The slowdown is likely to be realated to a few specific debug options such as debugging for mutexes, spinlocks, lists, etc. I guess that the most interesting information would be a difference in the debug options between the general kernel and the debug kernel. Hopefully we can put together a set of debug options that are cause the test to run over slow.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 5 lines · Cyril Hrubis
Hi!
Quoted 3 lines
Well, we have not achieved a reliable way to detect debug kernels in LTP. While I looking at our RHEL9 kernel config file. The general kernel also enables things like "
CONFIG_DEBUG_KERNEL=y".
Quoted 7 lines
The slowdown is likely to be realated to a few specific debug options such as debugging for mutexes, spinlocks, lists, etc. I guess that the most interesting information would be a difference in the debug options between the general kernel and the debug kernel. Hopefully we can put together a set of debug options that are cause the test to run over slow.
I have carefully compared the differences between the general kernel config-file and the debug kernel config-file.
Below are some configurations that are only enabled in the debug kernel and may cause kernel performance degradation.
The rough thoughts I have is to create a SET for those configurations, If the SUT kernel maps some of them, we reset the timeout using the value multiplier obtained from calibration.
e.g. if mapped N number of the configs we use (timeout * N) as the max_runtime.
Or next, we extract this method to the whole LTP timeout setting if possible?
#Lock debugging:CONFIG_PROVE_LOCKINGCONFIG_LOCKDEPCONFIG_DEBUG_SPINLOCK
#Mutex debuggingCONFIG_DEBUG_RT_MUTEXES=yCONFIG_DEBUG_MUTEXES=y
#Memory debugging:CONFIG_DEBUG_PAGEALLOCCONFIG_KASANCONFIG_SLUB_RCU_DEBUG
#Tracing and profiling:CONFIG_TRACE_IRQFLAGSCONFIG_LATENCYTOPCONFIG_DEBUG_NET
#Filesystem debugging:CONFIG_EXT4_DEBUGCONFIG_QUOTA_DEBUG
#Miscellaneous debugging:CONFIG_FAULT_INJECTIONCONFIG_DEBUG_OBJECTS
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH 1/1 lib: add TST_DYNAMICAL_RUNTIME option
Hi, this is a draft patch to reflect the method came up in the thread, if people agree I will polish it and send a complete one later.
--- include/tst_kconfig.h | 44 +++++++++++++++++++++++++++++++++++++++++++ include/tst_test.h | 1 + include/tst_timer.h | 30 +++++++++++++++++++++++++++++ lib/tst_test.c | 12 ++++++++---- 4 files changed, 83 insertions(+), 4 deletions(-)
include/tst_kconfig.h +44 -0
@@ -98,4 +98,48 @@ struct tst_kcmdline_var {*/ void tst_kcmdline_parse (struct tst_kcmdline_var params[],size_t params_len); +/* +* List of debug-related kernel config options that may degrade performance when enabled. +*/ +static const char *const tst_kconf_debug_options[][2 ] = { +/* Lock debugging */ + {"CONFIG_PROVE_LOCKING=y" ,NULL }, + {"CONFIG_LOCKDEP=y" ,NULL }, + {"CONFIG_DEBUG_SPINLOCK=y" ,NULL }, + +/* Mutexes debugging */ + {"CONFIG_DEBUG_RT_MUTEXES=y" ,NULL }, + {"CONFIG_DEBUG_MUTEXES=y" ,NULL }, + +/* Memory debugging */ + {"CONFIG_DEBUG_PAGEALLOC=y" ,NULL }, + {"CONFIG_KASAN=y" ,NULL }, + {"CONFIG_SLUB_RCU_DEBUG=y" ,NULL }, + +/* Tracing and profiling */ + {"CONFIG_TRACE_IRQFLAGS=y" ,NULL }, + {"CONFIG_LATENCYTOP=y" ,NULL }, + {"CONFIG_DEBUG_NET=y" ,NULL }, + +/* Filesystem debugging */ + {"CONFIG_EXT4_DEBUG=y" ,NULL }, + {"CONFIG_QUOTA_DEBUG=y" ,NULL }, + +/* Miscellaneous debugging */ + {"CONFIG_FAULT_INJECTION=y" ,NULL }, + {"CONFIG_DEBUG_OBJECTS=y" ,NULL }, + + {NULL ,NULL }/* End of the array */ +}; + +static inline int tst_kconfig_debug_matches (void ) +{ +int i, num =1 ; + +for (i =0 ; tst_kconf_debug_options[i][0 ] !=NULL ; i++) + num +=tst_kconfig_check (tst_kconf_debug_options[i]); + +return num; +} +#endif /* TST_KCONFIG_H__ */
include/tst_test.h +1 -0
@@ -235,6 +235,7 @@ struct tst_tag {extern unsigned int tst_variant;#define TST_UNLIMITED_RUNTIME (-1) +#define TST_DYNAMICAL_RUNTIME (-2)/** * struct tst_ulimit_val - An ulimit resource and value.
include/tst_timer.h +30 -0
@@ -17,6 +17,7 @@#include <mqueue.h> #include <time.h> #include "tst_test.h" +#include "tst_clocks.h" #include "lapi/common_timers.h" #include "lapi/posix_types.h" #include "lapi/syscalls.h"
@@ -1074,4 +1075,33 @@ static inline long long tst_timer_elapsed_us(void) return tst_timespec_to_us(tst_timer_elapsed()); } +#define CALLIBRATE_LOOPS 120000000 + +/* +* Measures the time taken by the CPU to perform a specified +* number of empty loops for calibration. +*/ +static inline int tst_callibrate (void ) +{ +int i; +struct timespec start, stop; +long long diff; + +for (i =0 ; i <CALLIBRATE_LOOPS ; i++) + __asm__ __volatile__ ("" :"+g" (i) : :); + +tst_clock_gettime (CLOCK_MONOTONIC_RAW , &start); + +for (i =0 ; i <CALLIBRATE_LOOPS ; i++) + __asm__ __volatile__ ("" :"+g" (i) : :); + +tst_clock_gettime (CLOCK_MONOTONIC_RAW , &stop); + + diff =tst_timespec_diff_us (stop, start); + +tst_res (TINFO ,"CPU did %i loops in %llius" ,CALLIBRATE_LOOPS , diff); + +return diff; +} +#endif /* TST_TIMER */
lib/tst_test.c +8 -4
@@ -1265,8 +1265,8 @@ static void do_setup(int argc, char *argv[])if (!tst_test)tst_brk (TBROK ,"No tests to run" ); -if (tst_test->max_runtime <-1 ) { -tst_brk (TBROK ,"Invalid runtime value %i" , +if (tst_test->max_runtime <-2 ) { +tst_brk (TBROK ,"Invalid runtime value %d" , results->max_runtime); }
@@ -1695,7 +1695,6 @@ unsigned int tst_remaining_runtime(void)return 0 ; } -unsigned int tst_multiply_timeout (unsigned int timeout) {parse_mul (&timeout_mul,"LTP_TIMEOUT_MUL" ,0.099 ,10000 );
@@ -1715,8 +1714,13 @@ static void set_timeout(void)return ; } +if (results->max_runtime ==TST_DYNAMICAL_RUNTIME ) { +tst_res (TINFO ,"Timeout is decited in running time" ); + results->max_runtime = (tst_callibrate () /1000 ) *tst_kconfig_debug_matches (); + } +if (results->max_runtime <0 ) { -tst_brk (TBROK ,"max_runtime must to be >= -1! (%d)" , +tst_brk (TBROK ,"max_runtime must to be >= -2! (%d)" , results->max_runtime); }
Signature · 5 lines
2.47.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [Draft PATCH] lib: add TST_DYNAMICAL_RUNTIME option
Modify starvation.c by adding ".max_runtime = TST_DYNAMICAL_RUNTIME".
Test on general kernel:
# ./starvation tst_tmpdir.c:316: TINFO: Using /tmp/LTP_stanNtz1s as tmpdir (xfs filesystem) tst_test.c:1894: TINFO: LTP version: 20240930 tst_test.c:1898: TINFO: Tested kernel: 6.12.0-30.el10.ppc64le #1 SMP Tue Nov 19 13:50:01 EST 2024 ppc64le tst_test.c:1718: TINFO: Timeout is decided in running time ../include/tst_timer.h:1102: TINFO: CPU did 120000000 loops in 61797us tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.12.0-30.el10.ppc64le/config' tst_kconfig.c:531: TINFO: Constraint 'CONFIG_PROVE_LOCKING=y' not satisfied! tst_kconfig.c:477: TINFO: Variables: tst_kconfig.c:495: TINFO: CONFIG_PROVE_LOCKING=n tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.12.0-30.el10.ppc64le/config' tst_kconfig.c:531: TINFO: Constraint 'CONFIG_LOCKDEP=y' not satisfied! tst_kconfig.c:477: TINFO: Variables: tst_kconfig.c:486: TINFO: CONFIG_LOCKDEP Undefined ... tst_test.c:1729: TINFO: Timeout per run is 0h 01m 31s starvation.c:98: TINFO: Setting affinity to CPU 0 starvation.c:146: TPASS: Haven't reproduced scheduler starvation.
Summary:
passed 1 failed 0 broken 0 skipped 0
warnings 0
Test on debug-kernel:
# ./starvation tst_tmpdir.c:316: TINFO: Using /tmp/LTP_staVKoH2k as tmpdir (xfs filesystem) tst_test.c:1898: TINFO: LTP version: 20240930 tst_test.c:1902: TINFO: Tested kernel: 6.12.0-30.el10.ppc64le+debug #1 SMP Tue Nov 19 13:41:20 EST 2024 ppc64le tst_test.c:1718: TINFO: Timeout is decided in running time ../include/tst_timer.h:1102: TINFO: CPU did 120000000 loops in 68663us tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.12.0-30.el10.ppc64le+debug/config' tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.12.0-30.el10.ppc64le+debug/config' ... tst_test.c:1733: TINFO: Timeout per run is 0h 18m 38s starvation.c:71: TINFO: Setting affinity to CPU 0 starvation.c:116: TPASS: Haven't reproduced scheduler starvation.
Summary:
passed 1 failed 0 broken 0 skipped 0
warnings 0
include/tst_kconfig.h struct tst_kcmdline_var { · line 140
Quoted 7 lines · Li Wang
include/tst_kconfig.h +6
-0+static inline int tst_kconfig_debug_matches (void ) +{ +int i, num =1 ; + +for (i =0 ; tst_kconf_debug_options[i][0 ] !=NULL ; i++) + num +=tst_kconfig_check (tst_kconf_debug_options[i]);
This should be: num += !tst_kconfig_check(tst_kconf_debug_options[i]);
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 10 lines
I have carefully compared the differences between the general kernel config-file and the debug kernel config-file.
Below are some configurations that are only enabled in the debug kernel and may cause kernel performance degradation.
The rough thoughts I have is to create a SET for those configurations, If the SUT kernel maps some of them, we reset the timeout using the value multiplier obtained from calibration.
Quoted 5 lines
e.g. if mapped N number of the configs we use (timeout * N) as the
max_runtime.Or next, we extract this method to the whole LTP timeout setting if possible?
That actually sounds good to me, if we detect certain kernel options that are know to slow down the process execution it makes a good sense to multiply the timeouts for all tests directly in the test library.
Quoted 26 lines
#Lock debugging:
CONFIG_PROVE_LOCKINGCONFIG_LOCKDEPCONFIG_DEBUG_SPINLOCK#Mutex debugging
CONFIG_DEBUG_RT_MUTEXES=yCONFIG_DEBUG_MUTEXES=y#Memory debugging:
CONFIG_DEBUG_PAGEALLOCCONFIG_KASANCONFIG_SLUB_RCU_DEBUG#Tracing and profiling:
CONFIG_TRACE_IRQFLAGSCONFIG_LATENCYTOPCONFIG_DEBUG_NET#Filesystem debugging:
CONFIG_EXT4_DEBUGCONFIG_QUOTA_DEBUG#Miscellaneous debugging:
CONFIG_FAULT_INJECTIONCONFIG_DEBUG_OBJECTS
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 14 lines · Cyril Hrubis
Hi!
Quoted 13 lines
I have carefully compared the differences between the general kernel config-file and the debug kernel config-file.
Below are some configurations that are only enabled in the debug kernel and may cause kernel performance degradation.
The rough thoughts I have is to create a SET for those configurations, If the SUT kernel maps some of them, we reset the timeout using the value multiplier obtained from calibration.
e.g. if mapped N number of the configs we use (timeout * N) as the
max_runtime.
Quoted 7 lines
Quoted 2 lines
Or next, we extract this method to the whole LTP timeout setting if possible?
That actually sounds good to me, if we detect certain kernel options that are know to slow down the process execution it makes a good sense to multiply the timeouts for all tests directly in the test library.
Thanks.
After thinking it over, I guess we'd better _only_ apply this method to some special slow tests (aka. more easily timeout tests). If we do the examination of those kernel options in the library for all, that maybe a burden to most quick tests, which always finish in a few seconds (far less than the default 30s).
Therefore, I came up with a new option for .max_runtime, which is TST_DYNAMICAL_RUNTIME. Similar to the TST_UNLIMITED_RUNTIME we ever use. Test by adding this .max_runtime = TST_DYNAIMCAL_RUNTIME that will try to find a proper timeout value in the running time for the test.
See: https://lists.linux.it/pipermail/ltp/2024-November/040990.html
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 11 lines
After thinking it over, I guess we'd better _only_ apply this method to some special slow tests (aka. more easily timeout tests). If we do the examination of those kernel options in the library for all, that maybe a burden to most quick tests, which always finish in a few seconds (far less than the default 30s).
Therefore, I came up with a new option for .max_runtime, which is
TST_DYNAMICAL_RUNTIME. Similar to theTST_UNLIMITED_RUNTIMEwe ever use. Test by adding this .max_runtime =TST_DYNAIMCAL_RUNTIMEthat will try to find a proper timeout value in the running time for the test.
I was thinking to only multiply the max_runtime defined by the test in the library. That way only slow tests that set the max_runtime would be affected.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 13 lines · Cyril Hrubis
Hi!
Quoted 11 lines
After thinking it over, I guess we'd better _only_ apply this method to some special slow tests (aka. more easily timeout tests). If we do the examination of those kernel options in the library for all, that maybe a burden to most quick tests, which always finish in a few seconds (far less than the default 30s).
Therefore, I came up with a new option for .max_runtime, which is
TST_DYNAMICAL_RUNTIME. Similar to theTST_UNLIMITED_RUNTIMEwe ever use. Test by adding this .max_runtime =TST_DYNAIMCAL_RUNTIMEthat will try to find a proper timeout value in the running time for the test.
Quoted 4 lines
I was thinking to only multiply the
max_runtimedefined by the test in the library. That way only slow tests that set themax_runtimewould be affected.
Ok, that also indicates the test is slower. I will apply that to non-zero '.max_runtime'
tests and resend a patch. Thanks!
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp