Skip to contentSkip to conversation

sched_football: harden kickoff synchronization on high-CPU systems

PATCH In review Jan Polensky 2 participants 7 messages Feb 24 – Mar 2

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched_football: harden kickoff synchronization on high-CPU systems

The sched_football test has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.

At game start the referee can transition into kickoff while not all defense threads have reliably reached their blocking path yet. This creates a window where offense threads may run and increment the_ball.

Fix the startup protocol:

- Introduce a dedicated defense_ready_barrier to rendezvous all defense threads with the referee before kickoff.
- Increase the pre-kickoff settle time (RT: 100 ms, non-RT: 2.5 s) to account for large CPU counts and loaded systems.
- Add a compiler barrier in the defense busy-loop to prevent it from being optimized away.
- Add an additional barrier step after the initial start phase so all threads are positioned deterministically before the game begins.

---
 .../func/sched_football/sched_football.c      | 23 ++++++++++++++-----
 1 file changed, 17 insertions(+), 6 deletions(-)
testcases/realtime/func/sched_football/sched_football.c +17 -6
index 2cb85322d782..08cdc2fd8b4e 100644
@@ -50,6 +50,7 @@ static tst_atomic_t game_over;
 static char *str_game_length;
 static char *str_players_per_team;
 static pthread_barrier_t start_barrier;
+static pthread_barrier_t defense_ready_barrier;
 
 /* These are fans running across the field. They're trying to interrupt/distract everyone */
 void *thread_fan(void *arg LTP_ATTRIBUTE_UNUSED)
@@ -81,11 +82,13 @@ void *thread_defense(void *arg LTP_ATTRIBUTE_UNUSED)
 {
 	prctl(PR_SET_NAME, "defense", 0, 0, 0);
 	pthread_barrier_wait(&start_barrier);
+	pthread_barrier_wait(&defense_ready_barrier);
 	while (!tst_atomic_load(&kickoff_flag))
 		;
 
-	/*keep the ball from being moved */
+	/* Keep the ball from being moved using a compiler barrier */
 	while (!tst_atomic_load(&game_over)) {
+		__asm__ __volatile__("" ::: "memory");
 	}
 
 	return NULL;
@@ -124,14 +127,18 @@ void referee(int game_length)
 	/* Start the game! */
 	atrace_marker_write("sched_football", "Game_started!");
 	pthread_barrier_wait(&start_barrier);
-	usleep(200000);
+
+	/* Wait for defense to be ready before starting the game */
+	pthread_barrier_wait(&defense_ready_barrier);
+
+	/* Give defense threads time to establish */
+	if (tst_check_preempt_rt())
+		usleep(100000);
+	else
+		usleep(2500000);
 
 	tst_atomic_store(0, &the_ball);
 	tst_atomic_store(1, &kickoff_flag);
-	if (tst_check_preempt_rt())
-		usleep(20000);
-	else
-		usleep(2000000);
 
 	/* Watch the game */
 	while ((now.tv_sec - start.tv_sec) < game_length) {
@@ -170,6 +177,9 @@ static void do_test(void)
 	sched_setscheduler(0, SCHED_FIFO, &param);
 	tst_atomic_store(0, &kickoff_flag);
 
+	/* Defense ready barrier: defense threads + referee */
+	pthread_barrier_init(&defense_ready_barrier, NULL, players_per_team + 1);
+
 	/*
 	 * Start the offense
 	 * They are lower priority than defense, so they must be started first.
@@ -197,6 +207,7 @@ static void do_test(void)
 	referee(game_length);
 
 	pthread_barrier_destroy(&start_barrier);
+	pthread_barrier_destroy(&defense_ready_barrier);
 }
 
 static void do_setup(void)
Signature · 5 lines
2.53.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

Hi!

It looks good to me, but I'm wondering if we can use TST_CHECKPOINT_WAIT2() in here to sync multiple threads instead of using pthrea barriers which are a bit more complex.

Kind regards,

Signature · 7 lines
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw
In reply to Jan Polensky
Quoted 5 lines · Jan Polensky

The sched_football test has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.

I've not had time to closely review your suggestion here, but it sounds reasonable.

That said, I want to warn you and ensure you are aware: the RT_PUSH_IPI feature in the scheduler breaks the RT invariant sched_football is testing.

I see this as a bug with that feature, but the scalability RT_PUSH_IPI allows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.

So you might want to make sure you disable that feature before testing via:

# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/features

thanks
-john

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw
Quoted 21 lines · John Stultz via ltp
Quoted 5 lines · Jan Polensky

The sched_football test has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.

I've not had time to closely review your suggestion here, but it sounds reasonable.

That said, I want to warn you and ensure you are aware: the RT_PUSH_IPI feature in the scheduler breaks the RT invariant sched_football is testing.

I see this as a bug with that feature, but the scalability RT_PUSH_IPI allows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.

Quoted 5 lines

So you might want to make sure you disable that feature before testing via:

# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/features

thanks
-john

Thanks for your deep analysis on the possible issue. I'm not an RT expert, but I trust your expertise in here :-) Will leave this patch review to someone who's more skilled than me in this topic.

I have a suggestion tho.

About the NO_RT_PUSH_IPI, we are lucky: LTP provides a safe mechanism to set the sys configurations and to restore it to default value after test. You can find this in the struct tst_test and it's called .save_restore [1]

I think it's worth to force this option according to the underlying variant (and properly document this in the code with a comment).

WDYT?

[1] https://linux-test-project.readthedocs.io/en/latest/developers/api_c_tests.html#tst-test-definition

Signature · 7 lines
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
Quoted 39 lines · Andrea Cervesato
Quoted 21 lines · John Stultz via ltp
Quoted 5 lines · Jan Polensky

The sched_football test has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.

I've not had time to closely review your suggestion here, but it sounds reasonable.

That said, I want to warn you and ensure you are aware: the RT_PUSH_IPI feature in the scheduler breaks the RT invariant sched_football is testing.

I see this as a bug with that feature, but the scalability RT_PUSH_IPI allows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.

Quoted 5 lines

So you might want to make sure you disable that feature before testing via:

# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/features

thanks
-john

Thanks for your deep analysis on the possible issue. I'm not an RT expert, but I trust your expertise in here :-) Will leave this patch review to someone who's more skilled than me in this topic.

I have a suggestion tho.

About the NO_RT_PUSH_IPI, we are lucky: LTP provides a safe mechanism to set the sys configurations and to restore it to default value after test. You can find this in the struct tst_test and it's called .save_restore [1]

Quoted 4 lines

I think it's worth to force this option according to the underlying variant (and properly document this in the code with a comment).

WDYT?

That seems reasonable, as long as it's clearly labeled as a workaround and hopefully can be dropped (or kernel version limited) when the issue is finally addressed.

thanks
-john

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

Hi John,

Quoted 3 lines

That seems reasonable, as long as it's clearly labeled as a workaround and hopefully can be dropped (or kernel version limited) when the issue is finally addressed.

Sure, that can be eventually addressed by using tst_kvercmp() or tst_kvercmp2 (to be even more specific).

Kind regards,

Signature · 7 lines
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
In reply to John Stultz via ltp
Quoted 44 lines · John Stultz
Quoted 42 lines · Andrea Cervesato
Quoted 21 lines · John Stultz via ltp
Quoted 5 lines · Jan Polensky

The sched_football test has been intermittently failing, most noticeably on systems with many CPUs and/or under load, due to a startup ordering hole around kickoff.

I've not had time to closely review your suggestion here, but it sounds reasonable.

That said, I want to warn you and ensure you are aware: the RT_PUSH_IPI feature in the scheduler breaks the RT invariant sched_football is testing.

I see this as a bug with that feature, but the scalability RT_PUSH_IPI allows for seems more important to folks who are doing RT work then the mis-behavior. Steven and I talked awhile back about some ideas on how we might be able to do the pull in a more efficient way while still holding the invariant true, and I have a bug to track it, but its not been high enough priority to get bandwidth yet.

Quoted 5 lines

So you might want to make sure you disable that feature before testing via:

# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/features

thanks
-john

Thanks for your deep analysis on the possible issue. I'm not an RT expert, but I trust your expertise in here :-) Will leave this patch review to someone who's more skilled than me in this topic.

I have a suggestion tho.

About the NO_RT_PUSH_IPI, we are lucky: LTP provides a safe mechanism to set the sys configurations and to restore it to default value after test. You can find this in the struct tst_test and it's called .save_restore [1]

I think it's worth to force this option according to the underlying variant (and properly document this in the code with a comment).

Quoted 8 lines
Quoted 1 line

WDYT?

That seems reasonable, as long as it's clearly labeled as a workaround and hopefully can be dropped (or kernel version limited) when the issue is finally addressed.

thanks
-john

Hi Andrea, hi John,

thank you for the thorough review and the helpful remarks. After going through the feedback, I think it makes sense to step back and rework the patch. The main objective is to drive the failure rate down as much as possible, and the current version still shows weaknesses, especially with respect to steal time. On heavily loaded systems I also still observe frequent TBROK results, so the timing clearly needs further tuning.
I will take some time to revisit the design and incorporate these aspects before posting a revised version.
Thanks again for your comments and suggestions.

Thanks & Greetings
Jan

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp