Skip to contentSkip to conversation

PSCI: don't call CPU_OFF when it might fail

PATCH New Mark Rutland 3 participants 10 messages Sep 5, 2014 – Sep 8, 2014

Subsystems (2)
lore
Share & tools
lore

Mark Rutland

author reviewer
#0 lore raw

cover PSCI: don't call CPU_OFF when it might fail

[Apologies to all who received this already, I typo'd the LAKML address]

In certain conditions PSCI's CPU_OFF call may return DENIED (in the presence of a UP OS on the secure world), or might not exist (in early implementations prior to PSCI 0.2). Both the arm and arm64 callers fail to handle the former, and the arm caller doesn't handle the latter.

We can determine wether a CPU_OFF call will succeed by calling MIGRATE_INFO_TYPE. If we receive 2 (A.K.A. PSCI_0_2_TOS_MP) in response we know that CPU_OFF should succeed, and in other cases it is necessary to discover which CPU said OS lives on and attempt migration. As there don't seem to be any UP implementations out there so far, for now we can limit ourselves to only supporting CPU hotplug in the PSCI_0_2_TOS_MP case. Hotplug support on such systems will require additional work, but at least they won't blow up by default.

These patches add the necessary checks to the arm and arm64 PSCI callers to prevent the issues described above. MIGRATE_INFO_TYPE is a mandatory part of PSCI 0.2, so no conforming implementation should be adversely affected. Both the ARM Trusted Firmware and KVM respond to MIGRATE_INFO_TYPE with PSCI_0_2_TOS_MP, so should see no loss of functionality (verified on Juno).

Thanks,
Mark.

Mark Rutland (2):
arm64: psci: respect MIGRATE_INFO_TYPE
arm: psci: don't call CPU_OFF blindly

 arch/arm/kernel/psci_smp.c | 36 +++++++++++++++++++++++++++++-------
 arch/arm64/kernel/psci.c   | 14 ++++++++++++++
 2 files changed, 43 insertions(+), 7 deletions(-)
Signature · 1 line
1.9.1
author reviewer
#1 lore raw

PATCH 1/2 arm64: psci: respect MIGRATE_INFO_TYPE

Commit e71246a23acb (PSCI: Add initial support for PSCIv0.2 functions) added hooks for MIGRATE_INFO_TYPE, but didn't call it anywhere. This means that currently a UP Trusted OS can reject CPU_OFF calls, which will cause things to blow up.

This patch ensures we test MIGRATE_INFO_TYPE before calling CPU_OFF. If there's a UP trusted OS hotplug is rejected. Full support for PSCI in the presence of a UP trusted OS will require the use of MIGRATE and will have to come as a later patch.

Cc Ashwin Chaugule <[email protected]>
Cc Rob Herring <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Ian Campbell <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
---
 arch/arm64/kernel/psci.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/arm64/kernel/psci.c +14 -0
index 5539547..a5407af 100644
@@ -386,6 +386,20 @@ static int cpu_psci_cpu_disable(unsigned int cpu)
 	/* Fail early if we don't have CPU_OFF support */
 	if (!psci_ops.cpu_off)
 		return -EOPNOTSUPP;
+
+	/*
+	 * In the presence of a UP trusted OS, it might not be possible to
+	 * hotplug certain CPUs, and CPU_OFF may return (which would be bad).
+	 * Supporting a UP trusted OS requires careful use of
+	 * MIGRATE_INFO_UP_CPU and MIGRATE, so for now fail in the presence of
+	 * a UP Trusted OS.
+	 */
+	if (psci_ops.migrate_info_type &&
+		psci_ops.migrate_info_type() != PSCI_0_2_TOS_MP) {
+			pr_warn("Unable to handle UP trusted OS\n");
+			return -EPERM;
+	}
+
 	return 0;
 }
 
Signature · 1 line
1.9.1
#2 lore raw

PATCH 1/2 arm64: psci: respect MIGRATE_INFO_TYPE

Hello.

Quoted 32 lines · Mark Rutland

Commit e71246a23acb (PSCI: Add initial support for PSCIv0.2 functions) added hooks for MIGRATE_INFO_TYPE, but didn't call it anywhere. This means that currently a UP Trusted OS can reject CPU_OFF calls, which will cause things to blow up.

This patch ensures we test MIGRATE_INFO_TYPE before calling CPU_OFF. If there's a UP trusted OS hotplug is rejected. Full support for PSCI in the presence of a UP trusted OS will require the use of MIGRATE and will have to come as a later patch.

Cc Ashwin Chaugule <[email protected]>
Cc Rob Herring <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Ian Campbell <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
---
  arch/arm64/kernel/psci.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
arch/arm64/kernel/psci.c +2 -0
index 5539547..a5407af 100644
@@ -386,6 +386,20 @@ static int cpu_psci_cpu_disable(unsigned int cpu)
  	/* Fail early if we don't have CPU_OFF support */
  	if (!psci_ops.cpu_off)
  		return -EOPNOTSUPP;
+
+	/*

arch/arm64/kernel/psci.c static int cpu_psci_cpu_disable(unsigned int cpu) · line 400

Quoted 10 lines
arch/arm64/kernel/psci.c +10 -0
+	 * In the presence of a UP trusted OS, it might not be possible to
+	 * hotplug certain CPUs, and CPU_OFF may return (which would be bad).
+	 * Supporting a UP trusted OS requires careful use of
+	 * MIGRATE_INFO_UP_CPU and MIGRATE, so for now fail in the presence of
+	 * a UP Trusted OS.
+	 */
+	if (psci_ops.migrate_info_type &&
+		psci_ops.migrate_info_type() != PSCI_0_2_TOS_MP) {
+			pr_warn("Unable to handle UP trusted OS\n");
+			return -EPERM;

The above 2 lines are indented too much to the right.

arch/arm64/kernel/psci.c static int cpu_psci_cpu_disable(unsigned int cpu) · line 402

Quoted 2 lines
arch/arm64/kernel/psci.c +2 -0
+	}
+

WNR, Sergei

author reviewer
#3 lore raw

PATCH 1/2 arm64: psci: respect MIGRATE_INFO_TYPE

[...]

Quoted 6 lines

arch/arm64/kernel/psci.c static int cpu_psci_cpu_disable(unsigned int cpu) · line 400

Quoted 4 lines
arch/arm64/kernel/psci.c +4 -0
+	if (psci_ops.migrate_info_type &&
+		psci_ops.migrate_info_type() != PSCI_0_2_TOS_MP) {
+			pr_warn("Unable to handle UP trusted OS\n");
+			return -EPERM;

The above 2 lines are indented too much to the right.

Noted; locally fixed. Thanks.

The logic is the same regardless, so I won't send out a v2 unless someone has an issue with that logic, rather than the style.

Mark.

author reviewer
#4 lore raw

PATCH 2/2 arm: psci: don't call CPU_OFF blindly

In reply to Mark Rutland

The generic PSCI operations for arm check the presence of a CPU_OFF ID far too late, and in the absence of an ID will panic(), rather than producing a warning.

This patch adds a psci_cpu_disable callback which tests the presence of a CPU_OFF id. As this is called earlier than psci_cpu_die, the failure can be handled gracefully without brining down the system. Additionally a check is added for a UP trusted OS in the presence of PSCI 0.2+. Full support will require the use of MIGRATE, but for now rejecting hotplug will prevent psci_cpu_die from brining down the system.

The now redundant check for scpi_ops.cpu_off is removed from psci_cpu_die. At the same time, the whitespace is corrected from seven spaces to tabs.

Cc Ashwin Chaugule <[email protected]>
Cc Rob Herring <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Ian Campbell <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
---
 arch/arm/kernel/psci_smp.c | 36 +++++++++++++++++++++++++++++-------
 1 file changed, 29 insertions(+), 7 deletions(-)

Stefano, I've followed your lead with the __ref annotation here, but I couldn't figure out why they exist on cpu_die and cpu_kill; it feels rather dodgy. Do you know why they were added, or if they are superfluous?

There are some other cleanups that should happen here (static, CPU_METHOD_OF_DECLARE), but those will come as a later cleanups series.

arch/arm/kernel/psci_smp.c +29 -7
index 28a1db4..2b00d3c 100644
@@ -56,17 +56,38 @@ static int psci_boot_secondary(unsigned int cpu, struct task_struct *idle)
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
+int __ref psci_cpu_disable(unsigned int cpu)
+{
+	/* Fail early if we don't have CPU_OFF support */
+	if (!psci_ops.cpu_off)
+		return -EOPNOTSUPP;
+
+	/*
+	 * In the presence of a UP trusted OS, it might not be possible to
+	 * hotplug certain CPUs, and CPU_OFF may return (which would be bad).
+	 * Supporting a UP trusted OS requires careful use of
+	 * MIGRATE_INFO_UP_CPU and MIGRATE, so for now fail in the presence of
+	 * a UP Trusted OS.
+	 */
+	if (psci_ops.migrate_info_type &&
+		psci_ops.migrate_info_type() != PSCI_0_2_TOS_MP) {
+			pr_warn("Unable to handle UP trusted OS\n");
+			return -EPERM;
+	}
+
+	return 0;
+}
+
 void __ref psci_cpu_die(unsigned int cpu)
 {
-       const struct psci_power_state ps = {
-               .type = PSCI_POWER_STATE_TYPE_POWER_DOWN,
-       };
+	const struct psci_power_state ps = {
+		.type = PSCI_POWER_STATE_TYPE_POWER_DOWN,
+	};
 
-       if (psci_ops.cpu_off)
-               psci_ops.cpu_off(ps);
+	psci_ops.cpu_off(ps);
 
-       /* We should never return */
-       panic("psci: cpu %d failed to shutdown\n", cpu);
+	/* We should never return */
+	panic("psci: cpu %d failed to shutdown\n", cpu);
 }
 
 int __ref psci_cpu_kill(unsigned int cpu)
@@ -109,6 +130,7 @@ bool __init psci_smp_available(void)
 struct smp_operations __initdata psci_smp_ops = {
 	.smp_boot_secondary	= psci_boot_secondary,
 #ifdef CONFIG_HOTPLUG_CPU
+	.cpu_disable		= psci_cpu_disable,
 	.cpu_die		= psci_cpu_die,
 	.cpu_kill		= psci_cpu_kill,
 #endif
Signature · 1 line
1.9.1
#5 lore raw

PATCH 2/2 arm: psci: don't call CPU_OFF blindly

Quoted 23 lines · Mark Rutland

The generic PSCI operations for arm check the presence of a CPU_OFF ID far too late, and in the absence of an ID will panic(), rather than producing a warning.

This patch adds a psci_cpu_disable callback which tests the presence of a CPU_OFF id. As this is called earlier than psci_cpu_die, the failure can be handled gracefully without brining down the system. Additionally a check is added for a UP trusted OS in the presence of PSCI 0.2+. Full support will require the use of MIGRATE, but for now rejecting hotplug will prevent psci_cpu_die from brining down the system.

The now redundant check for scpi_ops.cpu_off is removed from psci_cpu_die. At the same time, the whitespace is corrected from seven spaces to tabs.

Cc Ashwin Chaugule <[email protected]>
Cc Rob Herring <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Ian Campbell <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
Quoted 7 lines
---
 arch/arm/kernel/psci_smp.c | 36 +++++++++++++++++++++++++++++-------
 1 file changed, 29 insertions(+), 7 deletions(-)

Stefano, I've followed your lead with the __ref annotation here, but I couldn't figure out why they exist on cpu_die and cpu_kill; it feels rather dodgy. Do you know why they were added, or if they are superfluous?

I don't think that __ref is needed.
That particular snipped of code came from Rob Herring, maybe he knows why it was added in the first place.

Quoted 64 lines

There are some other cleanups that should happen here (static, CPU_METHOD_OF_DECLARE), but those will come as a later cleanups series.

arch/arm/kernel/psci_smp.c +29 -7
index 28a1db4..2b00d3c 100644
@@ -56,17 +56,38 @@ static int psci_boot_secondary(unsigned int cpu, struct task_struct *idle)
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
+int __ref psci_cpu_disable(unsigned int cpu)
+{
+	/* Fail early if we don't have CPU_OFF support */
+	if (!psci_ops.cpu_off)
+		return -EOPNOTSUPP;
+
+	/*
+	 * In the presence of a UP trusted OS, it might not be possible to
+	 * hotplug certain CPUs, and CPU_OFF may return (which would be bad).
+	 * Supporting a UP trusted OS requires careful use of
+	 * MIGRATE_INFO_UP_CPU and MIGRATE, so for now fail in the presence of
+	 * a UP Trusted OS.
+	 */
+	if (psci_ops.migrate_info_type &&
+		psci_ops.migrate_info_type() != PSCI_0_2_TOS_MP) {
+			pr_warn("Unable to handle UP trusted OS\n");
+			return -EPERM;
+	}
+
+	return 0;
+}
+
 void __ref psci_cpu_die(unsigned int cpu)
 {
-       const struct psci_power_state ps = {
-               .type = PSCI_POWER_STATE_TYPE_POWER_DOWN,
-       };
+	const struct psci_power_state ps = {
+		.type = PSCI_POWER_STATE_TYPE_POWER_DOWN,
+	};
 
-       if (psci_ops.cpu_off)
-               psci_ops.cpu_off(ps);
+	psci_ops.cpu_off(ps);
 
-       /* We should never return */
-       panic("psci: cpu %d failed to shutdown\n", cpu);
+	/* We should never return */
+	panic("psci: cpu %d failed to shutdown\n", cpu);
 }
 
 int __ref psci_cpu_kill(unsigned int cpu)
@@ -109,6 +130,7 @@ bool __init psci_smp_available(void)
 struct smp_operations __initdata psci_smp_ops = {
 	.smp_boot_secondary	= psci_boot_secondary,
 #ifdef CONFIG_HOTPLUG_CPU
+	.cpu_disable		= psci_cpu_disable,
 	.cpu_die		= psci_cpu_die,
 	.cpu_kill		= psci_cpu_kill,
 #endif
Signature · 1 line
1.9.1
author reviewer
#6 lore raw

PATCH 2/2 arm: psci: don't call CPU_OFF blindly

Quoted 28 lines · Stefano Stabellini
Quoted 27 lines · Mark Rutland

The generic PSCI operations for arm check the presence of a CPU_OFF ID far too late, and in the absence of an ID will panic(), rather than producing a warning.

This patch adds a psci_cpu_disable callback which tests the presence of a CPU_OFF id. As this is called earlier than psci_cpu_die, the failure can be handled gracefully without brining down the system. Additionally a check is added for a UP trusted OS in the presence of PSCI 0.2+. Full support will require the use of MIGRATE, but for now rejecting hotplug will prevent psci_cpu_die from brining down the system.

The now redundant check for scpi_ops.cpu_off is removed from psci_cpu_die. At the same time, the whitespace is corrected from seven spaces to tabs.

Cc Ashwin Chaugule <[email protected]>
Cc Rob Herring <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Ian Campbell <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
---
 arch/arm/kernel/psci_smp.c | 36 +++++++++++++++++++++++++++++-------
 1 file changed, 29 insertions(+), 7 deletions(-)
Quoted 7 lines
Quoted 3 lines

Stefano, I've followed your lead with the __ref annotation here, but I couldn't figure out why they exist on cpu_die and cpu_kill; it feels rather dodgy. Do you know why they were added, or if they are superfluous?

I don't think that __ref is needed.
That particular snipped of code came from Rob Herring, maybe he knows why it was added in the first place.

I've traced that back, but can't see any rationale. Perhaps that had something to do with __cpuinit/__cpuexit, but there doesn't seem to be any reason for them now. I guess the __ref annotation on cpu_die in arch/arm/kernel/smp.c can go too given __cpuinit and __cpuexit are gone.

Rob, do you have any idea either way?

Thanks,
Mark.

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2013-March/158570.html

#7 lore raw

cover PSCI: don't call CPU_OFF when it might fail

In reply to Mark Rutland
Quoted 25 lines · Mark Rutland

[Apologies to all who received this already, I typo'd the LAKML address]

In certain conditions PSCI's CPU_OFF call may return DENIED (in the presence of a UP OS on the secure world), or might not exist (in early implementations prior to PSCI 0.2). Both the arm and arm64 callers fail to handle the former, and the arm caller doesn't handle the latter.

We can determine wether a CPU_OFF call will succeed by calling MIGRATE_INFO_TYPE. If we receive 2 (A.K.A. PSCI_0_2_TOS_MP) in response we know that CPU_OFF should succeed, and in other cases it is necessary to discover which CPU said OS lives on and attempt migration. As there don't seem to be any UP implementations out there so far, for now we can limit ourselves to only supporting CPU hotplug in the PSCI_0_2_TOS_MP case. Hotplug support on such systems will require additional work, but at least they won't blow up by default.

These patches add the necessary checks to the arm and arm64 PSCI callers to prevent the issues described above. MIGRATE_INFO_TYPE is a mandatory part of PSCI 0.2, so no conforming implementation should be adversely affected. Both the ARM Trusted Firmware and KVM respond to MIGRATE_INFO_TYPE with PSCI_0_2_TOS_MP, so should see no loss of functionality (verified on Juno).

Thanks,
Mark.

Quoted 7 lines

Mark Rutland (2):
arm64: psci: respect MIGRATE_INFO_TYPE
arm: psci: don't call CPU_OFF blindly

 arch/arm/kernel/psci_smp.c | 36 +++++++++++++++++++++++++++++-------
 arch/arm64/kernel/psci.c   | 14 ++++++++++++++
 2 files changed, 43 insertions(+), 7 deletions(-)

The patches look good to me and AFACT they would work fine on Xen 4.4 and the soon to be release 4.5.

author reviewer
#8 lore raw

cover PSCI: don't call CPU_OFF when it might fail

Quoted 30 lines · Stefano Stabellini
Quoted 29 lines · Mark Rutland

[Apologies to all who received this already, I typo'd the LAKML address]

In certain conditions PSCI's CPU_OFF call may return DENIED (in the presence of a UP OS on the secure world), or might not exist (in early implementations prior to PSCI 0.2). Both the arm and arm64 callers fail to handle the former, and the arm caller doesn't handle the latter.

We can determine wether a CPU_OFF call will succeed by calling MIGRATE_INFO_TYPE. If we receive 2 (A.K.A. PSCI_0_2_TOS_MP) in response we know that CPU_OFF should succeed, and in other cases it is necessary to discover which CPU said OS lives on and attempt migration. As there don't seem to be any UP implementations out there so far, for now we can limit ourselves to only supporting CPU hotplug in the PSCI_0_2_TOS_MP case. Hotplug support on such systems will require additional work, but at least they won't blow up by default.

These patches add the necessary checks to the arm and arm64 PSCI callers to prevent the issues described above. MIGRATE_INFO_TYPE is a mandatory part of PSCI 0.2, so no conforming implementation should be adversely affected. Both the ARM Trusted Firmware and KVM respond to MIGRATE_INFO_TYPE with PSCI_0_2_TOS_MP, so should see no loss of functionality (verified on Juno).

Thanks,
Mark.

Mark Rutland (2):
arm64: psci: respect MIGRATE_INFO_TYPE
arm: psci: don't call CPU_OFF blindly

Quoted 6 lines
Quoted 3 lines
 arch/arm/kernel/psci_smp.c | 36 +++++++++++++++++++++++++++++-------
 arch/arm64/kernel/psci.c   | 14 ++++++++++++++
 2 files changed, 43 insertions(+), 7 deletions(-)

The patches look good to me and AFACT they would work fine on Xen 4.4 and the soon to be release 4.5.

Cheers.

Can I take that as an ack?

Mark.

#9 lore raw

cover PSCI: don't call CPU_OFF when it might fail

Quoted 35 lines · Mark Rutland
Quoted 34 lines · Stefano Stabellini
Quoted 32 lines · Mark Rutland

[Apologies to all who received this already, I typo'd the LAKML address]

In certain conditions PSCI's CPU_OFF call may return DENIED (in the presence of a UP OS on the secure world), or might not exist (in early implementations prior to PSCI 0.2). Both the arm and arm64 callers fail to handle the former, and the arm caller doesn't handle the latter.

We can determine wether a CPU_OFF call will succeed by calling MIGRATE_INFO_TYPE. If we receive 2 (A.K.A. PSCI_0_2_TOS_MP) in response we know that CPU_OFF should succeed, and in other cases it is necessary to discover which CPU said OS lives on and attempt migration. As there don't seem to be any UP implementations out there so far, for now we can limit ourselves to only supporting CPU hotplug in the PSCI_0_2_TOS_MP case. Hotplug support on such systems will require additional work, but at least they won't blow up by default.

These patches add the necessary checks to the arm and arm64 PSCI callers to prevent the issues described above. MIGRATE_INFO_TYPE is a mandatory part of PSCI 0.2, so no conforming implementation should be adversely affected. Both the ARM Trusted Firmware and KVM respond to MIGRATE_INFO_TYPE with PSCI_0_2_TOS_MP, so should see no loss of functionality (verified on Juno).

Thanks,
Mark.

Mark Rutland (2):
arm64: psci: respect MIGRATE_INFO_TYPE
arm: psci: don't call CPU_OFF blindly

 arch/arm/kernel/psci_smp.c | 36 +++++++++++++++++++++++++++++-------
 arch/arm64/kernel/psci.c   | 14 ++++++++++++++
 2 files changed, 43 insertions(+), 7 deletions(-)
Quoted 6 lines
Quoted 2 lines

The patches look good to me and AFACT they would work fine on Xen 4.4 and the soon to be release 4.5.

Cheers.

Can I take that as an ack?

Yes