sched_football: harden kickoff synchronization on high-CPU systems
PATCH 1/1 sched_football: harden kickoff synchronization on high-CPU systems
The sched_football test has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.
At game start the referee can transition into kickoff while not all defense threads have reliably reached their blocking path yet. This creates a window where offense threads may run and increment the_ball.
Fix the startup protocol:
- Introduce a dedicated defense_ready_barrier to rendezvous all defense threads with the referee before kickoff.
- Increase the pre-kickoff settle time (RT: 100 ms, non-RT: 2.5 s) to account for large CPU counts and loaded systems.
- Add a compiler barrier in the defense busy-loop to prevent it from being optimized away.
- Add an additional barrier step after the initial start phase so all threads are positioned deterministically before the game begins.
--- .../func/sched_football/sched_football.c | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-)
testcases/realtime/func/sched_football/sched_football.c +17 -6
@@ -50,6 +50,7 @@ static tst_atomic_t game_over;static char *str_game_length;static char *str_players_per_team;static pthread_barrier_t start_barrier; +static pthread_barrier_t defense_ready_barrier;/* These are fans running across the field. They're trying to interrupt/distract everyone */ void *thread_fan(void *arg LTP_ATTRIBUTE_UNUSED)
@@ -81,11 +82,13 @@ void *thread_defense(void *arg LTP_ATTRIBUTE_UNUSED) {prctl (PR_SET_NAME ,"defense" ,0 ,0 ,0 );pthread_barrier_wait (&start_barrier); +pthread_barrier_wait (&defense_ready_barrier);while (!tst_atomic_load (&kickoff_flag)) ; -/*keep the ball from being moved */ +/* Keep the ball from being moved using a compiler barrier */ while (!tst_atomic_load (&game_over)) { + __asm__ __volatile__("" :::"memory" ); }return NULL ;
@@ -124,14 +127,18 @@ void referee(int game_length)/* Start the game! */ atrace_marker_write ("sched_football" ,"Game_started!" );pthread_barrier_wait (&start_barrier); -usleep (200000 ); + +/* Wait for defense to be ready before starting the game */ +pthread_barrier_wait (&defense_ready_barrier); + +/* Give defense threads time to establish */ +if (tst_check_preempt_rt ()) +usleep (100000 ); +else +usleep (2500000 );tst_atomic_store (0 , &the_ball);tst_atomic_store (1 , &kickoff_flag); -if (tst_check_preempt_rt ()) -usleep (20000 ); -else -usleep (2000000 );/* Watch the game */ while ((now.tv_sec - start.tv_sec) < game_length) {
@@ -170,6 +177,9 @@ static void do_test(void)sched_setscheduler (0 ,SCHED_FIFO , ¶m);tst_atomic_store (0 , &kickoff_flag); +/* Defense ready barrier: defense threads + referee */ +pthread_barrier_init (&defense_ready_barrier,NULL , players_per_team +1 ); +/* * Start the offense * They are lower priority than defense, so they must be started first.
@@ -197,6 +207,7 @@ static void do_test(void)referee (game_length);pthread_barrier_destroy (&start_barrier); +pthread_barrier_destroy (&defense_ready_barrier); }static void do_setup (void )
Signature · 5 lines
2.53.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
It looks good to me, but I'm wondering if we can use TST_CHECKPOINT_WAIT2() in here to sync multiple threads instead of using pthrea barriers which are a bit more complex.
Kind regards,
Signature · 7 lines
Andrea Cervesato SUSE QE Automation Engineer Linux [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 5 lines · Jan Polensky
The
sched_footballtest has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.
I've not had time to closely review your suggestion here, but it sounds reasonable.
That said, I want to warn you and ensure you are aware: the RT_PUSH_IPI feature in the scheduler breaks the RT invariant sched_football is testing.
I see this as a bug with that feature, but the scalability RT_PUSH_IPI allows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.
So you might want to make sure you disable that feature before testing via:
# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/features
thanks
-john
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 21 lines · John Stultz via ltp
Quoted 5 lines · Jan Polensky
The
sched_footballtest has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.I've not had time to closely review your suggestion here, but it sounds reasonable.
That said, I want to warn you and ensure you are aware: the
RT_PUSH_IPIfeature in the scheduler breaks the RT invariantsched_footballis testing.I see this as a bug with that feature, but the scalability
RT_PUSH_IPIallows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.
Quoted 5 lines
So you might want to make sure you disable that feature before testing via:
# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/featuresthanks
-john
Thanks for your deep analysis on the possible issue. I'm not an RT expert, but I trust your expertise in here :-) Will leave this patch review to someone who's more skilled than me in this topic.
I have a suggestion tho.
About the NO_RT_PUSH_IPI, we are lucky: LTP provides a safe mechanism to set the sys configurations and to restore it to default value after test. You can find this in the struct tst_test and it's called .save_restore [1]
I think it's worth to force this option according to the underlying variant (and properly document this in the code with a comment).
WDYT?
[1] https://linux-test-project.readthedocs.io/en/latest/developers/api_c_tests.html#tst-test-definition
Signature · 7 lines
Andrea Cervesato SUSE QE Automation Engineer Linux [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 39 lines · Andrea Cervesato
Quoted 21 lines · John Stultz via ltp
Quoted 5 lines · Jan Polensky
The
sched_footballtest has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.I've not had time to closely review your suggestion here, but it sounds reasonable.
That said, I want to warn you and ensure you are aware: the
RT_PUSH_IPIfeature in the scheduler breaks the RT invariantsched_footballis testing.I see this as a bug with that feature, but the scalability
RT_PUSH_IPIallows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.Quoted 5 lines
So you might want to make sure you disable that feature before testing via:
# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/featuresthanks
-johnThanks for your deep analysis on the possible issue. I'm not an RT expert, but I trust your expertise in here :-) Will leave this patch review to someone who's more skilled than me in this topic.
I have a suggestion tho.
About the
NO_RT_PUSH_IPI, we are lucky: LTP provides a safe mechanism to set the sys configurations and to restore it to default value after test. You can find this in thestruct tst_testand it's called.save_restore[1]
Quoted 4 lines
I think it's worth to force this option according to the underlying variant (and properly document this in the code with a comment).
WDYT?
That seems reasonable, as long as it's clearly labeled as a workaround and hopefully can be dropped (or kernel version limited) when the issue is finally addressed.
thanks
-john
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi John,
Quoted 3 lines
That seems reasonable, as long as it's clearly labeled as a workaround and hopefully can be dropped (or kernel version limited) when the issue is finally addressed.
Sure, that can be eventually addressed by using tst_kvercmp() or tst_kvercmp2 (to be even more specific).
Kind regards,
Signature · 7 lines
Andrea Cervesato SUSE QE Automation Engineer Linux [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 44 lines · John Stultz
Quoted 42 lines · Andrea Cervesato
Quoted 21 lines · John Stultz via ltp
Quoted 5 lines · Jan Polensky
The
sched_footballtest has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.I've not had time to closely review your suggestion here, but it sounds reasonable.
That said, I want to warn you and ensure you are aware: the
RT_PUSH_IPIfeature in the scheduler breaks the RT invariantsched_footballis testing.I see this as a bug with that feature, but the scalability
RT_PUSH_IPIallows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.Quoted 5 lines
So you might want to make sure you disable that feature before testing via:
# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/featuresthanks
-johnThanks for your deep analysis on the possible issue. I'm not an RT expert, but I trust your expertise in here :-) Will leave this patch review to someone who's more skilled than me in this topic.
I have a suggestion tho.
About the
NO_RT_PUSH_IPI, we are lucky: LTP provides a safe mechanism to set the sys configurations and to restore it to default value after test. You can find this in thestruct tst_testand it's called.save_restore[1]I think it's worth to force this option according to the underlying variant (and properly document this in the code with a comment).
Quoted 8 lines
Quoted 1 line
WDYT?
That seems reasonable, as long as it's clearly labeled as a workaround and hopefully can be dropped (or kernel version limited) when the issue is finally addressed.
thanks
-john
Hi Andrea, hi John,
thank you for the thorough review and the helpful remarks. After going through the feedback, I think it makes sense to step back and rework the patch. The main objective is to drive the failure rate down as much as possible, and the current version still shows weaknesses, especially with respect to steal time. On heavily loaded systems I also still observe frequent TBROK results, so the timing clearly needs further tuning.
I will take some time to revisit the design and incorporate these aspects before posting a revised version.
Thanks again for your comments and suggestions.
Thanks & Greetings
Jan
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp