Skip to contentSkip to conversation

kernel/isolation: Asset that a housekeeping CPU comes up at boot time

PATCH Superseded Nicholas Piggin 4 participants 7 messages Jun 1, 2019 – Jun 25, 2019 Scheduler

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kernel/isolation: Asset that a housekeeping CPU comes up at boot time

With the change to allow the boot CPU0 to be isolated, it is possible to specify command line options that result in no housekeeping CPU online at boot.

An 8 CPU system booted with "nohz_full=0-6 maxcpus=4", for example.

It is not easily possible at housekeeping init time to know all the various SMP options that will result in an invalid configuration, so this patch adds a sanity check after SMP init, to ensure that a housekeeping CPU has been onlined.

The panic is undesirable, but it's better than the alternative of an obscure non deterministic failure. The panic will reliably happen when advanced parameters are used incorrectly.

---
 kernel/sched/isolation.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
kernel/sched/isolation.c +23 -0
index 123ea07a3f3b..7b9e1e0d4ec3 100644
@@ -63,6 +63,29 @@ void __init housekeeping_init(void)
 	WARN_ON_ONCE(cpumask_empty(housekeeping_mask));
 }
 
+static int __init housekeeping_verify_smp(void)
+{
+	int cpu;
+
+	/*
+	 * Early housekeeping setup is done before CPUs come up, and there are
+	 * a range of options scattered around that can restrict which CPUs
+	 * come up. It is possible to pass in a combination of housekeeping
+	 * and SMP arguments that result in housekeeping assigned to an
+	 * offline CPU.
+	 *
+	 * Check that condition here after SMP comes up, and give a useful
+	 * error message rather than an obscure non deterministic crash or
+	 * hang later.
+	 */
+	for_each_online_cpu(cpu) {
+		if (cpumask_test_cpu(cpu, housekeeping_mask))
+			return 0;
+	}
+	panic("Housekeeping: nohz_full= or isolcpus= resulted in no online CPUs for housekeeping.\n");
+}
+core_initcall(housekeeping_verify_smp);
+
 static int __init housekeeping_setup(char *str, enum hk_flags flags)
 {
 	cpumask_var_t non_housekeeping_mask;
Signature · 1 line
2.20.1
#1 lore raw

Nicholas Piggin's on June 1, 2019 9:39 pm:

Quoted 6 lines

With the change to allow the boot CPU0 to be isolated, it is possible to specify command line options that result in no housekeeping CPU online at boot.

An 8 CPU system booted with "nohz_full=0-6 maxcpus=4", for example.

Quoted 8 lines

It is not easily possible at housekeeping init time to know all the various SMP options that will result in an invalid configuration, so this patch adds a sanity check after SMP init, to ensure that a housekeeping CPU has been onlined.

The panic is undesirable, but it's better than the alternative of an obscure non deterministic failure. The panic will reliably happen when advanced parameters are used incorrectly.

Ping on this one? This should resolve Frederic's remaining objection to the series (at least until he solves it more generally).

As the series has already been merged, should we get this upstream before release?

Thanks,
Nick

maintainer
#2 lore raw
Quoted 16 lines · Nicholas Piggin

Nicholas Piggin's on June 1, 2019 9:39 pm:

Quoted 6 lines

With the change to allow the boot CPU0 to be isolated, it is possible to specify command line options that result in no housekeeping CPU online at boot.

An 8 CPU system booted with "nohz_full=0-6 maxcpus=4", for example.

Quoted 8 lines

It is not easily possible at housekeeping init time to know all the various SMP options that will result in an invalid configuration, so this patch adds a sanity check after SMP init, to ensure that a housekeeping CPU has been onlined.

The panic is undesirable, but it's better than the alternative of an obscure non deterministic failure. The panic will reliably happen when advanced parameters are used incorrectly.

Quoted 5 lines

Ping on this one? This should resolve Frederic's remaining objection to the series (at least until he solves it more generally).

As the series has already been merged, should we get this upstream before release?

I was hoping for feedback from Frederic, lacking that, I've queued it now.

#3 lore raw
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Nicholas Piggin

Nicholas Piggin's on June 1, 2019 9:39 pm:

Quoted 14 lines

With the change to allow the boot CPU0 to be isolated, it is possible to specify command line options that result in no housekeeping CPU online at boot.

An 8 CPU system booted with "nohz_full=0-6 maxcpus=4", for example.

It is not easily possible at housekeeping init time to know all the various SMP options that will result in an invalid configuration, so this patch adds a sanity check after SMP init, to ensure that a housekeeping CPU has been onlined.

The panic is undesirable, but it's better than the alternative of an obscure non deterministic failure. The panic will reliably happen when advanced parameters are used incorrectly.

Quoted 9 lines
Quoted 5 lines

Ping on this one? This should resolve Frederic's remaining objection to the series (at least until he solves it more generally).

As the series has already been merged, should we get this upstream before release?

I was hoping for feedback from Frederic, lacking that, I've queued it now.

Sorry I just came back from vacation. Any chance we can use a WARN() instead? I prefer to use panic() only when data is really threatened or such.

Thanks.

#4 lore raw

Frederic Weisbecker's on June 18, 2019 5:05 am:

Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Nicholas Piggin
Quoted 21 lines

Nicholas Piggin's on June 1, 2019 9:39 pm:

Quoted 6 lines

With the change to allow the boot CPU0 to be isolated, it is possible to specify command line options that result in no housekeeping CPU online at boot.

An 8 CPU system booted with "nohz_full=0-6 maxcpus=4", for example.

Quoted 8 lines

It is not easily possible at housekeeping init time to know all the various SMP options that will result in an invalid configuration, so this patch adds a sanity check after SMP init, to ensure that a housekeeping CPU has been onlined.

The panic is undesirable, but it's better than the alternative of an obscure non deterministic failure. The panic will reliably happen when advanced parameters are used incorrectly.

Ping on this one? This should resolve Frederic's remaining objection to the series (at least until he solves it more generally).

As the series has already been merged, should we get this upstream before release?

Quoted 6 lines
Quoted 3 lines

I was hoping for feedback from Frederic, lacking that, I've queued it now.

Sorry I just came back from vacation. Any chance we can use a WARN() instead? I prefer to use panic() only when data is really threatened or such.

I thought it was decided to panic here, because we don't assign a house keeping CPU so the system is unlikely to behave properly. A warn might scroll off the screen by the time things grind to a halt.

This is a one-time boot parameter misconfiguration, many cases of which can cause a panic and boot stop.

No question if we can make this more dynamic that would be better, but for near term at least can we go with this?

Thanks,
Nick

#5 lore raw
In reply to Nicholas Piggin
Quoted 38 lines · Nicholas Piggin

With the change to allow the boot CPU0 to be isolated, it is possible to specify command line options that result in no housekeeping CPU online at boot.

An 8 CPU system booted with "nohz_full=0-6 maxcpus=4", for example.

It is not easily possible at housekeeping init time to know all the various SMP options that will result in an invalid configuration, so this patch adds a sanity check after SMP init, to ensure that a housekeeping CPU has been onlined.

The panic is undesirable, but it's better than the alternative of an obscure non deterministic failure. The panic will reliably happen when advanced parameters are used incorrectly.

---
 kernel/sched/isolation.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
kernel/sched/isolation.c +10 -0
index 123ea07a3f3b..7b9e1e0d4ec3 100644
@@ -63,6 +63,29 @@ void __init housekeeping_init(void)
 	WARN_ON_ONCE(cpumask_empty(housekeeping_mask));
 }
 
+static int __init housekeeping_verify_smp(void)
+{
+	int cpu;
+
+	/*
+	 * Early housekeeping setup is done before CPUs come up, and there are
+	 * a range of options scattered around that can restrict which CPUs
+	 * come up. It is possible to pass in a combination of housekeeping
+	 * and SMP arguments that result in housekeeping assigned to an
+	 * offline CPU.

kernel/sched/isolation.c void __init housekeeping_init(void) · line 85

Quoted 10 lines
kernel/sched/isolation.c +10 -0
+	 *
+	 * Check that condition here after SMP comes up, and give a useful
+	 * error message rather than an obscure non deterministic crash or
+	 * hang later.
+	 */
+	for_each_online_cpu(cpu) {
+		if (cpumask_test_cpu(cpu, housekeeping_mask))
+			return 0;
+	}
+	panic("Housekeeping: nohz_full= or isolcpus= resulted in no online CPUs for housekeeping.\n");

I am hitting this panic when I boot my juno board.

I have CONFIG_CPU_ISOLATION=y but I don't pass nohuz_full nor isolcpus in the commandline. I think what's going on is that housekeeping_setup() doesn't get called and hence housekeeping_mask isn't initialized in my case, causing this check to fail and trigger the panic.

The below seems to 'fix' it though not sure if it's the right way forward. A revert obviously fixes it too but I doubt we want that :-)

kernel/sched/isolation.c +3 -0
index 7b9e1e0d4ec3..a9ca8628c1a2 100644
@@ -67,6 +67,9 @@ static int __init housekeeping_verify_smp(void)
 {
 	int cpu;
 
+	if (!housekeeping_flags)
+		return 0;
+
 	/*
 	 * Early housekeeping setup is done before CPUs come up, and there are
 	 * a range of options scattered around that can restrict which CPUs

Cheers

--
Qais Yousef

Quoted 9 lines
kernel/sched/isolation.c +3 -0
+}
+core_initcall(housekeeping_verify_smp);
+
 static int __init housekeeping_setup(char *str, enum hk_flags flags)
 {
 	cpumask_var_t non_housekeeping_mask;
Signature · 1 line
2.20.1
#6 lore raw

Qais Yousef's on June 24, 2019 8:57 pm:

Quoted 53 lines · Nicholas Piggin
Quoted 38 lines

With the change to allow the boot CPU0 to be isolated, it is possible to specify command line options that result in no housekeeping CPU online at boot.

An 8 CPU system booted with "nohz_full=0-6 maxcpus=4", for example.

It is not easily possible at housekeeping init time to know all the various SMP options that will result in an invalid configuration, so this patch adds a sanity check after SMP init, to ensure that a housekeeping CPU has been onlined.

The panic is undesirable, but it's better than the alternative of an obscure non deterministic failure. The panic will reliably happen when advanced parameters are used incorrectly.

---
 kernel/sched/isolation.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
kernel/sched/isolation.c +10 -0
index 123ea07a3f3b..7b9e1e0d4ec3 100644
@@ -63,6 +63,29 @@ void __init housekeeping_init(void)
 	WARN_ON_ONCE(cpumask_empty(housekeeping_mask));
 }
 
+static int __init housekeeping_verify_smp(void)
+{
+	int cpu;
+
+	/*
+	 * Early housekeeping setup is done before CPUs come up, and there are
+	 * a range of options scattered around that can restrict which CPUs
+	 * come up. It is possible to pass in a combination of housekeeping
+	 * and SMP arguments that result in housekeeping assigned to an
+	 * offline CPU.

kernel/sched/isolation.c void __init housekeeping_init(void) · line 85

Quoted 10 lines
kernel/sched/isolation.c +10 -0
+	 *
+	 * Check that condition here after SMP comes up, and give a useful
+	 * error message rather than an obscure non deterministic crash or
+	 * hang later.
+	 */
+	for_each_online_cpu(cpu) {
+		if (cpumask_test_cpu(cpu, housekeeping_mask))
+			return 0;
+	}
+	panic("Housekeeping: nohz_full= or isolcpus= resulted in no online CPUs for housekeeping.\n");

I am hitting this panic when I boot my juno board.

Quoted 7 lines

I have CONFIG_CPU_ISOLATION=y but I don't pass nohuz_full nor isolcpus in the commandline. I think what's going on is that housekeeping_setup() doesn't get called and hence housekeeping_mask isn't initialized in my case, causing this check to fail and trigger the panic.

The below seems to 'fix' it though not sure if it's the right way forward. A revert obviously fixes it too but I doubt we want that :-)

That'll do it. Thanks for the report and investigation.

Quoted 34 lines
kernel/sched/isolation.c +3 -0
index 7b9e1e0d4ec3..a9ca8628c1a2 100644
@@ -67,6 +67,9 @@ static int __init housekeeping_verify_smp(void)
 {
 	int cpu;
 
+	if (!housekeeping_flags)
+		return 0;
+
 	/*
 	 * Early housekeeping setup is done before CPUs come up, and there are
 	 * a range of options scattered around that can restrict which CPUs

Cheers

--
Qais Yousef

Quoted 9 lines
kernel/sched/isolation.c +3 -0
+}
+core_initcall(housekeeping_verify_smp);
+
 static int __init housekeeping_setup(char *str, enum hk_flags flags)
 {
 	cpumask_var_t non_housekeeping_mask;
Signature · 1 line
2.20.1