xen: Don't call panic if ARM TF cpu off returns DENIED
PATCH 1/1 xen: Don't call panic if ARM TF cpu off returns DENIED
From: Dmytro Semenets <[email protected]>
According to PSCI specification ARM TF can return DENIED on CPU OFF. This patch brings the hypervisor into compliance with the PSCI specification.
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
--- xen/arch/arm/psci.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
xen/arch/arm/psci.c +3 -2
@@ -63,8 +63,9 @@ void call_psci_cpu_off(void)/* If successfull the PSCI cpu_off call doesn't return */ arm_smccc_smc (PSCI_0_2_FN32_CPU_OFF , &res); -panic ("PSCI cpu off failed for CPU%d err=%d\n" ,smp_processor_id (), -PSCI_RET (res)); +if (PSCI_RET (res) !=PSCI_DENIED ) +panic ("PSCI cpu off failed for CPU%d err=%d\n" ,smp_processor_id (), +PSCI_RET (res)); } }
Signature · 1 line
2.25.1
Hi,
Quoted 3 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]>
According to PSCI specification ARM TF can return DENIED on CPU OFF.
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
Also, did you reproduce on HW? If so, on which CPU this will fail?
Quoted 2 lines
This patch brings the hypervisor into compliance with the PSCI specification.
Now it means CPU off will never be turned off using PSCI. Instead, we would end up to spin in Xen. This would be a problem because we could save less power.
Quoted 2 lines
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
Reading both 5.5.2 and 5.9.1 together, DENIED would be returned when the trusted OS can only run on one core.
Some of the trusted OS are migratable. So I think we should first attempt to migrate the CPU. Then if it doesn't work, we should prevent the CPU to go offline.
That said, upstream doesn't support cpu offlining (I don't know for your use case). In case of shutdown, it is not necessary to offline the CPU, so we could avoid to call CPU_OFF on all CPUs but one. Something like:
xen/arch/arm/shutdown.c +3 -5
@@ -8,7 +8,9 @@static void noreturnhalt_this_cpu (void *arg) { -stop_cpu (); +ASSERT (!local_irq_enable ()); +while (1 ) +wfi (); }void machine_halt (void )
@@ -21,10 +23,6 @@ void machine_halt(void)smp_call_function (halt_this_cpu,NULL ,0 );local_irq_disable (); -/* Wait at most another 10ms for all other CPUs to go offline. */ -while ( (num_online_cpus () >1 ) && (timeout-- >0 ) ) -mdelay (1 ); -/* This is mainly for PSCI-0.2, which does not return if success. */ call_psci_system_off ();
Quoted 2 lines
Signed-off-by Dmytro Semenets <[email protected]>Reviewed-by Volodymyr Babchuk <[email protected]>
I don't recall to see patch on the ML recently for this. So is this an internal review?
Quoted 10 lines
--- xen/arch/arm/psci.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)xen/arch/arm/psci.c +0
-0@@ -63,8 +63,9 @@ void call_psci_cpu_off(void)
xen/arch/arm/psci.c void call_psci_cpu_off(void) · line 71
Quoted 10 lines
xen/arch/arm/psci.c +3
-2/* If successfull the PSCI cpu_off call doesn't return */ arm_smccc_smc (PSCI_0_2_FN32_CPU_OFF , &res); -panic ("PSCI cpu off failed for CPU%d err=%d\n" ,smp_processor_id (), -PSCI_RET (res)); +if (PSCI_RET (res) !=PSCI_DENIED ) +panic ("PSCI cpu off failed for CPU%d err=%d\n" ,smp_processor_id (), +PSCI_RET (res)); } }
Cheers,
Signature · 1 line
Julien Grall
Hi Julien,
Quoted 9 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Quoted 3 lines
Also, did you reproduce on HW? If so, on which CPU this will fail?
Yes, we reproduced this on HW. In our case it failed on CPU0. To be fair, in our case it had nothing to do with Trusted OS. It is just platform limitation - it can't turn off CPU0. But from Xen perspective there is no difference - CPU_OFF call returns DENIED.
Quoted 6 lines
Quoted 2 lines
This patch brings the hypervisor into compliance with the PSCI specification.
Now it means CPU off will never be turned off using PSCI. Instead, we would end up to spin in Xen. This would be a problem because we could save less power.
Agreed.
Quoted 7 lines
Quoted 3 lines
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
Reading both 5.5.2 and 5.9.1 together, DENIED would be returned when the trusted OS can only run on one core.
Quoted 9 lines
Some of the trusted OS are migratable. So I think we should first attempt to migrate the CPU. Then if it doesn't work, we should prevent the CPU to go offline.
That said, upstream doesn't support cpu offlining (I don't know for your use case). In case of shutdown, it is not necessary to offline the CPU, so we could avoid to call
CPU_OFFon all CPUs but one. Something like:
This is even better approach yes. But you mentioned CPU_OFF. Did you mean SYSTEM_RESET?
Quoted 29 lines
xen/arch/arm/shutdown.c +3
-5@@ -8,7 +8,9 @@static void noreturnhalt_this_cpu (void *arg) { -stop_cpu (); +ASSERT (!local_irq_enable ()); +while (1 ) +wfi (); }void machine_halt (void )@@ -21,10 +23,6 @@ void machine_halt(void)smp_call_function (halt_this_cpu,NULL ,0 );local_irq_disable (); -/* Wait at most another 10ms for all other CPUs to go offline. */ -while ( (num_online_cpus () >1 ) && (timeout-- >0 ) ) -mdelay (1 ); -/* This is mainly for PSCI-0.2, which does not return if success. */ call_psci_system_off ();Quoted 2 lines
Signed-off-by Dmytro Semenets <[email protected]>Reviewed-by Volodymyr Babchuk <[email protected]>
Quoted 2 lines
I don't recall to see patch on the ML recently for this. So is this an internal review?
Yeah, sorry about that. Dmytro is a new member in our team and he is not yet familiar with differences in internal reviews and reviews in ML.
If you are interested, we had internal review at [1]:
[1] https://github.com/xen-troops/xen/pull/184
Quoted 22 lines
Quoted 10 lines
--- xen/arch/arm/psci.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)xen/arch/arm/psci.c +0
-0@@ -63,8 +63,9 @@ void call_psci_cpu_off(void)/* If successfull the PSCI cpu_off call doesn't return xen/arch/arm/psci.c void call_psci_cpu_off(void) · line 71
Quoted 10 lines
xen/arch/arm/psci.c +3
-2*/ arm_smccc_smc (PSCI_0_2_FN32_CPU_OFF , &res); -panic ("PSCI cpu off failed for CPU%d err=%d\n" ,smp_processor_id (), -PSCI_RET (res)); +if (PSCI_RET (res) !=PSCI_DENIED ) +panic ("PSCI cpu off failed for CPU%d err=%d\n" ,smp_processor_id (), +PSCI_RET (res)); } }Cheers,
Signature · 1 line
Volodymyr Babchuk at EPAM
Quoted 2 lines · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 6 lines
Quoted 3 lines · Julien Grall
Hi,
On 16/06/2022 14:55, [email protected] wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that CPU_OFF could return DENIED because of the firmware. Do you have a pointer to the spec?
Quoted 9 lines
Quoted 4 lines
Also, did you reproduce on HW? If so, on which CPU this will fail?
Yes, we reproduced this on HW. In our case it failed on CPU0. To be fair, in our case it had nothing to do with Trusted OS. It is just platform limitation - it can't turn off CPU0. But from Xen perspective there is no difference -
CPU_OFFcall returns DENIED.
Thanks for the clarification. I think I have seen that in the wild also but it never got on top of my queue. It is good that we are fixing it.
Quoted 21 lines
Quoted 7 lines
Quoted 3 lines
This patch brings the hypervisor into compliance with the PSCI specification.
Now it means CPU off will never be turned off using PSCI. Instead, we would end up to spin in Xen. This would be a problem because we could save less power.
Agreed.
Quoted 11 lines
Quoted 3 lines
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
Reading both 5.5.2 and 5.9.1 together, DENIED would be returned when the trusted OS can only run on one core.
Some of the trusted OS are migratable. So I think we should first attempt to migrate the CPU. Then if it doesn't work, we should prevent the CPU to go offline.
Quoted 8 lines
Quoted 5 lines
That said, upstream doesn't support cpu offlining (I don't know for your use case). In case of shutdown, it is not necessary to offline the CPU, so we could avoid to call
CPU_OFFon all CPUs but one. Something like:This is even better approach yes. But you mentioned
CPU_OFF. Did you meanSYSTEM_RESET?
By CPU_OFF I was referring to the fact that Xen will issue the call all CPUs but one. The remaining CPU will issue the command to reset/shutdown the system.
Quoted 12 lines
Quoted 12 lines
void
machine_halt(void)xen/arch/arm/shutdown.c +0
-4@@ -21,10 +23,6 @@ void machine_halt(void)smp_call_function (halt_this_cpu,NULL ,0 );local_irq_disable (); -/* Wait at most another 10ms for all other CPUs to go offline. */ -while ( (num_online_cpus () >1 ) && (timeout-- >0 ) ) -mdelay (1 ); -/* This is mainly for PSCI-0.2, which does not return if success. */ call_psci_system_off ();
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Signed-off-by Dmytro Semenets <[email protected]>Reviewed-by Volodymyr Babchuk <[email protected]>I don't recall to see patch on the ML recently for this. So is this an internal review?
Yeah, sorry about that. Dmytro is a new member in our team and he is not yet familiar with differences in internal reviews and reviews in ML.
No worries. I usually classify internal review anything that was done privately. This looks to be a public review, althought not on xen-devel.
I understand that not all some of the patches are still in PoC stage and doing the review on your github is a good idea. But for those are meant to be for upstream (e.g. bug fixes, small patches), I would suggest to do the review on xen-devel directly.
Cheers,
Signature · 1 line
Julien Grall
чт, 16 июн. 2022 г. в 22:09, Julien Grall <[email protected]>:
Quoted 21 lines · Volodymyr Babchuk
Quoted 2 lines
Hi Julien,
Hi Volodymyr,
Quoted 13 lines
Quoted 9 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
Quoted 6 lines
Quoted 1 line
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?
Actually CPU_OFF performed by Trusted OS. But Trusted OS is called by the ARM TF.
ARM TF for our platform doesn't call Trusted OS for CPU0 and returns DENIED instead.
Quoted 70 lines
Quoted 10 lines
Quoted 5 lines
Also, did you reproduce on HW? If so, on which CPU this will fail?
Yes, we reproduced this on HW. In our case it failed on CPU0. To be fair, in our case it had nothing to do with Trusted OS. It is just platform limitation - it can't turn off CPU0. But from Xen perspective there is no difference -
CPU_OFFcall returns DENIED.Thanks for the clarification. I think I have seen that in the wild also but it never got on top of my queue. It is good that we are fixing it.
Quoted 21 lines
Quoted 7 lines
Quoted 3 lines
This patch brings the hypervisor into compliance with the PSCI specification.
Now it means CPU off will never be turned off using PSCI. Instead, we would end up to spin in Xen. This would be a problem because we could save less power.
Agreed.
Quoted 11 lines
Quoted 3 lines
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
Reading both 5.5.2 and 5.9.1 together, DENIED would be returned when the trusted OS can only run on one core.
Some of the trusted OS are migratable. So I think we should first attempt to migrate the CPU. Then if it doesn't work, we should prevent the CPU to go offline.
Quoted 8 lines
Quoted 5 lines
That said, upstream doesn't support cpu offlining (I don't know for your use case). In case of shutdown, it is not necessary to offline the CPU, so we could avoid to call
CPU_OFFon all CPUs but one. Something like:This is even better approach yes. But you mentioned
CPU_OFF. Did you meanSYSTEM_RESET?By
CPU_OFFI was referring to the fact that Xen will issue the call all CPUs but one. The remaining CPU will issue the command to reset/shutdown the system.Quoted 10 lines
xen/arch/arm/shutdown.c void machine_halt(void) · line 26
Quoted 10 lines
void
machine_halt(void)xen/arch/arm/shutdown.c +0
-4@@ -21,10 +23,6 @@ void machine_halt(void)smp_call_function (halt_this_cpu,NULL ,0 );local_irq_disable (); -/* Wait at most another 10ms for all other CPUs to go offline. */ -while ( (num_online_cpus () >1 ) && (timeout-- >0 ) ) -mdelay (1 ); -/* This is mainly for PSCI-0.2, which does not return if success. */
Quoted 10 lines
Quoted 7 lines
xen/arch/arm/shutdown.c +0
-0call_psci_system_off ();Quoted 2 lines
Signed-off-by Dmytro Semenets <[email protected]>Reviewed-by Volodymyr Babchuk <[email protected]>I don't recall to see patch on the ML recently for this. So is this an internal review?
Yeah, sorry about that. Dmytro is a new member in our team and he is not yet familiar with differences in internal reviews and reviews in ML.
Quoted 8 lines
No worries. I usually classify internal review anything that was done privately. This looks to be a public review, althought not on xen-devel.
I understand that not all some of the patches are still in PoC stage and doing the review on your github is a good idea. But for those are meant to be for upstream (e.g. bug fixes, small patches), I would suggest to do the review on xen-devel directly
Sorry about that
Quoted 6 lines
Cheers,
--
Julien Grall
Hi Julien,
Quoted 13 lines · Julien Grall
Quoted 1 line · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 8 lines
Quoted 6 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?
Ah, looks like we are talking about different things. Indeed, CPU_OFF can return DENIED only because of Trusted OS. But entity that *returns* the error to a caller is a firmware.
Quoted 38 lines
Quoted 8 lines
Quoted 4 lines
Also, did you reproduce on HW? If so, on which CPU this will fail?
Yes, we reproduced this on HW. In our case it failed on CPU0. To be fair, in our case it had nothing to do with Trusted OS. It is just platform limitation - it can't turn off CPU0. But from Xen perspective there is no difference -
CPU_OFFcall returns DENIED.Thanks for the clarification. I think I have seen that in the wild also but it never got on top of my queue. It is good that we are fixing it.
Quoted 25 lines
Quoted 7 lines
Quoted 3 lines
This patch brings the hypervisor into compliance with the PSCI specification.
Now it means CPU off will never be turned off using PSCI. Instead, we would end up to spin in Xen. This would be a problem because we could save less power.
Agreed.
Quoted 16 lines
Quoted 3 lines
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
Reading both 5.5.2 and 5.9.1 together, DENIED would be returned when the trusted OS can only run on one core.
Some of the trusted OS are migratable. So I think we should first attempt to migrate the CPU. Then if it doesn't work, we should prevent the CPU to go offline.
That said, upstream doesn't support cpu offlining (I don't know for your use case). In case of shutdown, it is not necessary to offline the CPU, so we could avoid to call
CPU_OFFon all CPUs but one. Something like:
Quoted 7 lines
Quoted 2 lines
This is even better approach yes. But you mentioned
CPU_OFF. Did you meanSYSTEM_RESET?By
CPU_OFFI was referring to the fact that Xen will issue the call all CPUs but one. The remaining CPU will issue the command to reset/shutdown the system.
I just want to clarify: change that you suggested removes call to stop_cpu() in halt_this_cpu(). So no CPU_OFF will be sent at all.
All CPUs except one will spin in
while ( 1 )
wfi();while last cpu will issue SYSTEM_OFF or SYSTEM_RESET.
Is this correct?
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
void
machine_halt(void)xen/arch/arm/shutdown.c +0
-4@@ -21,10 +23,6 @@ void machine_halt(void)smp_call_function (halt_this_cpu,NULL ,0 );local_irq_disable (); -/* Wait at most another 10ms for all other CPUs to go offline. */ -while ( (num_online_cpus () >1 ) && (timeout-- >0 ) ) -mdelay (1 ); -/* This is mainly for PSCI-0.2, which does not return if success. */ call_psci_system_off ();Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Signed-off-by Dmytro Semenets <[email protected]>Reviewed-by Volodymyr Babchuk <[email protected]>I don't recall to see patch on the ML recently for this. So is this an internal review?
Yeah, sorry about that. Dmytro is a new member in our team and he is not
yet familiar with differences in internal reviews and reviews in ML.
Quoted 8 lines
No worries. I usually classify internal review anything that was done privately. This looks to be a public review, althought not on xen-devel.
I understand that not all some of the patches are still in PoC stage and doing the review on your github is a good idea. But for those are meant to be for upstream (e.g. bug fixes, small patches), I would suggest to do the review on xen-devel directly.
It not always clear if patch is eligible for upstream. At first we thought that problem is platform-specific and we weren't sure that we will find a proper upstreamable fix. Probably you saw that PR's name quite differs from final patch. This is because initial solution was completely different from final one.
Signature · 1 line
Volodymyr Babchuk at EPAM
Hi,
Quoted 20 lines · Volodymyr Babchuk
Quoted 18 lines · Julien Grall
Quoted 1 line · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 12 lines
Quoted 9 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Quoted 7 lines
Quoted 3 lines
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?Ah, looks like we are talking about different things. Indeed,
CPU_OFFcan return DENIED only because of Trusted OS. But entity that *returns* the error to a caller is a firmware.
Right, the interesting part is *why* DENIED is returned not *who* returns it.
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
Quoted 2 lines
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
Reading both 5.5.2 and 5.9.1 together, DENIED would be returned when the trusted OS can only run on one core.
Quoted 9 lines
Some of the trusted OS are migratable. So I think we should first attempt to migrate the CPU. Then if it doesn't work, we should prevent the CPU to go offline.
That said, upstream doesn't support cpu offlining (I don't know for your use case). In case of shutdown, it is not necessary to offline the CPU, so we could avoid to call
CPU_OFFon all CPUs but one. Something like:This is even better approach yes. But you mentioned
CPU_OFF. Did you meanSYSTEM_RESET?
Quoted 7 lines
Quoted 4 lines
By
CPU_OFFI was referring to the fact that Xen will issue the call all CPUs but one. The remaining CPU will issue the command to reset/shutdown the system.I just want to clarify: change that you suggested removes call to
stop_cpu()inhalt_this_cpu(). So noCPU_OFFwill be sent at all.
I was describing the existing behavior.
Quoted 9 lines
All CPUs except one will spin in
while ( 1 ) wfi();while last cpu will issue
SYSTEM_OFForSYSTEM_RESET.Is this correct?
Yes.
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
void
machine_halt(void)xen/arch/arm/shutdown.c +0
-4@@ -21,10 +23,6 @@ void machine_halt(void)smp_call_function (halt_this_cpu,NULL ,0 );local_irq_disable (); -/* Wait at most another 10ms for all other CPUs to go offline. */ -while ( (num_online_cpus () >1 ) && (timeout-- >0 ) ) -mdelay (1 ); -/* This is mainly for PSCI-0.2, which does not return if success. */ call_psci_system_off ();Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Signed-off-by Dmytro Semenets <[email protected]>Reviewed-by Volodymyr Babchuk <[email protected]>I don't recall to see patch on the ML recently for this. So is this an internal review?
Yeah, sorry about that. Dmytro is a new member in our team and he is not
yet familiar with differences in internal reviews and reviews in ML.No worries. I usually classify internal review anything that was done privately. This looks to be a public review, althought not on xen-devel.
Quoted 8 lines
Quoted 4 lines
I understand that not all some of the patches are still in PoC stage and doing the review on your github is a good idea. But for those are meant to be for upstream (e.g. bug fixes, small patches), I would suggest to do the review on xen-devel directly.
It not always clear if patch is eligible for upstream. At first we thought that problem is platform-specific and we weren't sure that we will find a proper upstreamable fix.
You can guess but not be sure until you send it to upstrema :). In fact,...
Quoted 3 lines
Probably you saw that PR's name
quite differs from final patch. This is because initial solution was completely different from final one.
... even before looking at your PR, this was the first solution I had in mind. I am still pondering whether this could be the best approach because I have the suspicion there might be some platform out relying on receiving the shutdown request on CPU0.
Anyway, this is so far just theorical, my proposal should solve your problem.
On a separate topic, the community is aiming to support a wide range of platforms out-of-the-box. I think platform-specific patches are acceptable so long they are self-contained (to some extend. I.e if you ask to support Xen on RPI3 then I would still probably argue against :)) or have a limited impact to the rest of the users (this is why we have alternative in Xen).
My point here is your initial solution may have been the preferred approach for upstream. So if you involve the community early, you are reducing the risk to have to backtrack and/or spend extra time in the wrong directions.
Cheers,
Signature · 1 line
Julien Grall
пт, 17 июн. 2022 г. в 14:12, Julien Grall <[email protected]>:
Hi Julien,
Quoted 28 lines
Hi,
Quoted 24 lines · Volodymyr Babchuk
Quoted 21 lines · Julien Grall
Quoted 1 line · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 4 lines
Quoted 2 lines · Julien Grall
Hi,
Quoted 8 lines
Quoted 7 lines · On 16/06/2022 14:55, [email protected] wrote:
Quoted 2 lines
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?
Quoted 6 lines
Quoted 3 lines
Ah, looks like we are talking about different things. Indeed,
CPU_OFFcan return DENIED only because of Trusted OS. But entity that *returns* the error to a caller is a firmware.Right, the interesting part is *why* DENIED is returned not *who* returns it.
ARM TF returns DENIED *only* for the platform I have.
We have a dissonance between spec and xen implementation because DENIED returned by
ARM TF or Thrusted OS or whatever is not a reason for panic. And we have issues with this.
If machine_restart() behaviour is more or less correct (sometimes reports about panic but restarts the machine)
but machine_halt() doesn't work at all.
Transit execution to CPU0 for my understanding is a workaround and this approach will fixmachine_restart() function but will not fix machine_halt(). Approach you suggested (spinning all cpus) will work but
will save less energy.
Quoted 105 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
Quoted 2 lines
Refer to "Arm Power State Coordination Interface (DEN0022D.b)" section 5.5.2
Reading both 5.5.2 and 5.9.1 together, DENIED would be returned when the trusted OS can only run on one core.
Quoted 9 lines
Some of the trusted OS are migratable. So I think we should first attempt to migrate the CPU. Then if it doesn't work, we should prevent the CPU to go offline.
That said, upstream doesn't support cpu offlining (I don't know for your use case). In case of shutdown, it is not necessary to offline the CPU, so we could avoid to call
CPU_OFFon all CPUs but one. Something like:This is even better approach yes. But you mentioned
CPU_OFF. Did you meanSYSTEM_RESET?Quoted 7 lines
Quoted 4 lines
By
CPU_OFFI was referring to the fact that Xen will issue the call all CPUs but one. The remaining CPU will issue the command to reset/shutdown the system.I just want to clarify: change that you suggested removes call to
stop_cpu()inhalt_this_cpu(). So noCPU_OFFwill be sent at all.I was describing the existing behavior.
Quoted 9 lines
All CPUs except one will spin in
while ( 1 ) wfi();while last cpu will issue
SYSTEM_OFForSYSTEM_RESET.Is this correct?
Yes.
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
void
machine_halt(void)xen/arch/arm/shutdown.c +0
-4@@ -21,10 +23,6 @@ void machine_halt(void)smp_call_function (halt_this_cpu,NULL ,0 );local_irq_disable (); -/* Wait at most another 10ms for all other CPUs to go offline. */ -while ( (num_online_cpus () >1 ) && (timeout-- >0 ) ) -mdelay (1 ); -/* This is mainly for PSCI-0.2, which does not return if success. */ call_psci_system_off ();Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Signed-off-by Dmytro Semenets <[email protected]>Reviewed-by Volodymyr Babchuk <[email protected]>I don't recall to see patch on the ML recently for this. So is this an internal review?
Yeah, sorry about that. Dmytro is a new member in our team and he is not
yet familiar with differences in internal reviews and reviews in ML.No worries. I usually classify internal review anything that was done privately. This looks to be a public review, althought not on xen-devel.
Quoted 8 lines
Quoted 4 lines
I understand that not all some of the patches are still in PoC stage and doing the review on your github is a good idea. But for those are meant to be for upstream (e.g. bug fixes, small patches), I would suggest to do the review on xen-devel directly.
It not always clear if patch is eligible for upstream. At first we thought that problem is platform-specific and we weren't sure that we will find a proper upstreamable fix.
You can guess but not be sure until you send it to upstrema :). In fact,...
Quoted 3 lines
Probably you saw that PR's name
quite differs from final patch. This is because initial solution was completely different from final one.... even before looking at your PR, this was the first solution I had in mind. I am still pondering whether this could be the best approach because I have the suspicion there might be some platform out relying on receiving the shutdown request on CPU0.
Anyway, this is so far just theorical, my proposal should solve your problem.
On a separate topic, the community is aiming to support a wide range of platforms out-of-the-box. I think platform-specific patches are acceptable so long they are self-contained (to some extend. I.e if you ask to support Xen on RPI3 then I would still probably argue against :)) or have a limited impact to the rest of the users (this is why we have alternative in Xen).
My point here is your initial solution may have been the preferred approach for upstream. So if you involve the community early, you are reducing the risk to have to backtrack and/or spend extra time in the wrong directions.
Cheers,
--
Julien Grall
Hi,
Quoted 34 lines · Dmytro Semenets
пт, 17 июн. 2022 г. в 14:12, Julien Grall <[email protected]>:
Hi Julien,Quoted 32 lines
Hi,
Quoted 27 lines · Volodymyr Babchuk
Quoted 13 lines · Julien Grall
Quoted 1 line · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 8 lines
Quoted 6 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?Ah, looks like we are talking about different things. Indeed,
CPU_OFFcan return DENIED only because of Trusted OS. But entity that *returns* the error to a caller is a firmware.
Quoted 6 lines
Quoted 2 lines
Right, the interesting part is *why* DENIED is returned not *who* returns it.
ARM TF returns DENIED *only* for the platform I have. We have a dissonance between spec and xen implementation because DENIED returned by
ARM TF or Thrusted OS or whatever is not a reason for panic.
I agree that's not a reason for panic. However, knowing the reason does help to figure out the correct approach.
For instance, one could have suggest to migrate the trusted OS to another pCPU. But this would not have worked for you because the DENIED is not about that.
Quoted 4 lines
And we
have issues with this.
Ifmachine_restart()behaviour is more or less correct (sometimes reports about panic but restarts the machine)
Right...
Quoted 1 line
but
machine_halt()doesn't work at al
... this should also be the case here because machine_halt() could also be called from cpu0. So I am a bit confused why you are saying it never works.
Quoted 3 lines
Transit execution to CPU0 for my understanding is a workaround and this approach will fix
machine_restart()function but will not fixmachine_halt().
I would say it is a more specific case of what the spec suggests (see below). But it should fix both machine_restart() and machine_halt() because the last CPU running will be CPU0. So Xen would call SYSTEM_* rather than CPU_OF. So I don't understand why you think it will fix one but not the other.
In fact, the idea to always run the request from a given CPU is quite similar to what the specification suggests (5.10.3 DEN0022D.b):
"
One way in which cores can be placed into a known state is to use calls to CPU_OFF on all online cores except for the last one, which instead uses SYSTEM_OFF. If a UP Trusted OS is present, this method only works if the core that calls SYSTEM_OFF is the one where the Trusted OS is resident, as calls to CPU_OFF on this core return a DENIED error. Any core can call SYSTEM_OFF.
"
For Xen, we would need to detect if the trusted OS is UP and where it is running. Then we could always restart/halt from that CPU or CPU0.
Quoted 3 lines
Approach
you suggested (spinning all cpus) will work but
will save less energy.
I am not sure to understand what's the concern about the energy here. From my understanding of the specification, SYSTEM_OFF will take care of switching off the power for all the cores. So at worse, the CPUs will spin for a few ms. This would like be more efficient than a call to PSCI CPU off.
This is different compare just turning off one CPU (i.e. CPU hot-unplug) because the CPU will end up to spin for a very long time. And this is why I wasn't OK with conditionally avoiding the panic.
Cheers,
Signature · 1 line
Julien Grall
Hi Julien,
Quoted 51 lines
Quoted 32 lines
Quoted 32 lines
Hi,
Quoted 27 lines · Volodymyr Babchuk
Quoted 13 lines · Julien Grall
Quoted 1 line · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 8 lines
Quoted 6 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?Ah, looks like we are talking about different things. Indeed,
CPU_OFFcan return DENIED only because of Trusted OS. But entity that *returns* the error to a caller is a firmware.Quoted 6 lines
Quoted 2 lines
Right, the interesting part is *why* DENIED is returned not *who* returns it.
ARM TF returns DENIED *only* for the platform I have. We have a dissonance between spec and xen implementation because DENIED returned by
ARM TF or Thrusted OS or whatever is not a reason for panic.I agree that's not a reason for panic. However, knowing the reason does help to figure out the correct approach.
For instance, one could have suggest to migrate the trusted OS to another pCPU. But this would not have worked for you because the DENIED is not about that.
Quoted 4 lines
And we
have issues with this.
Ifmachine_restart()behaviour is more or less correct (sometimes reports about panic but restarts the machine)
Quoted 6 lines
Right...
Quoted 1 line
but
machine_halt()doesn't work at al... this should also be the case here because
machine_halt()could also be called from cpu0. So I am a bit confused why you are saying it never works.
If machine_halt() called on a CPU other than CPU0 it caused panic and reboot. If it called on a CPU0 it also caused panic but after system power off and reboot
is not issued. In this state you can still call the xen console. But you can't reboot the system.
Quoted 10 lines
Quoted 4 lines
Transit execution to CPU0 for my understanding is a workaround and this approach will fix
machine_restart()function but will not fixmachine_halt().I would say it is a more specific case of what the spec suggests (see below). But it should fix both
machine_restart()andmachine_halt()because the last CPU running will be CPU0. So Xen would call SYSTEM_* rather thanCPU_OF. So I don't understand why you think it will fix one but not the other.
Looks like this is specific for my HW case. SYSTEM_OFF doesn't stop the whole system.
Quoted 35 lines
In fact, the idea to always run the request from a given CPU is quite similar to what the specification suggests (5.10.3 DEN0022D.b):
"
One way in which cores can be placed into a known state is to use calls toCPU_OFFon all online cores
except for the last one, which instead usesSYSTEM_OFF. If a UP Trusted OS is present, this method
only works if the core that callsSYSTEM_OFFis the one where the Trusted OS is resident, as calls toCPU_OFFon this core return a DENIED error. Any core can callSYSTEM_OFF. "For Xen, we would need to detect if the trusted OS is UP and where it is running. Then we could always restart/halt from that CPU or CPU0.
Quoted 3 lines
Approach
you suggested (spinning all cpus) will work but
will save less energy.I am not sure to understand what's the concern about the energy here.
From my understanding of the specification,SYSTEM_OFFwill take care
of switching off the power for all the cores. So at worse, the CPUs will spin for a few ms. This would like be more efficient than a call to PSCI CPU off.This is different compare just turning off one CPU (i.e. CPU hot-unplug) because the CPU will end up to spin for a very long time. And this is why I wasn't OK with conditionally avoiding the panic.
Cheers,
--
Julien Grall
Quoted 1 line · Dmytro Semenets
Hi Julien,
Hi Dmytro,
Quoted 54 lines
Quoted 54 lines
Quoted 32 lines
Quoted 32 lines
Hi,
Quoted 27 lines · Volodymyr Babchuk
Quoted 13 lines · Julien Grall
Quoted 1 line · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 8 lines
Quoted 6 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?Ah, looks like we are talking about different things. Indeed,
CPU_OFFcan return DENIED only because of Trusted OS. But entity that *returns* the error to a caller is a firmware.Quoted 6 lines
Quoted 2 lines
Right, the interesting part is *why* DENIED is returned not *who* returns it.
ARM TF returns DENIED *only* for the platform I have. We have a dissonance between spec and xen implementation because DENIED returned by
ARM TF or Thrusted OS or whatever is not a reason for panic.I agree that's not a reason for panic. However, knowing the reason does help to figure out the correct approach.
For instance, one could have suggest to migrate the trusted OS to another pCPU. But this would not have worked for you because the DENIED is not about that.
Quoted 4 lines
And we
have issues with this.
Ifmachine_restart()behaviour is more or less correct (sometimes reports about panic but restarts the machine)Right...
Quoted 1 line
but
machine_halt()doesn't work at al
Quoted 8 lines
Quoted 3 lines
... this should also be the case here because
machine_halt()could also be called from cpu0. So I am a bit confused why you are saying it never works.If
machine_halt()called on a CPU other than CPU0 it caused panic and reboot. If it called on a CPU0 it also caused panic but after system power off and reboot
is not issued. In this state you can still call the xen console. But you can't reboot the system.
I am lost. In a previous e-mail you said that PSCI CPU_OFF would return DENIED on CPU0. IOW, I understood that for other CPUs, it would succeed.
But here, you are tell me the opposite:
"If it called on a CPU0 it also caused panic but after system power off
and reboot".
If machine_halt() is called from CPU0, then CPU_OFF should not be called on CPU0. So where is that panic coming from?
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Transit execution to CPU0 for my understanding is a workaround and this approach will fix
machine_restart()function but will not fixmachine_halt().
Quoted 7 lines
Quoted 5 lines
I would say it is a more specific case of what the spec suggests (see below). But it should fix both
machine_restart()andmachine_halt()because the last CPU running will be CPU0. So Xen would call SYSTEM_* rather thanCPU_OF. So I don't understand why you think it will fix one but not the other.Looks like this is specific for my HW case.
SYSTEM_OFFdoesn't stop the whole system.
Hmmm... All the other CPUs should be off (or spinning with interrupt disabled), so are you saying that SYSTEM_OFF return?
Cheers,
Signature · 1 line
Julien Grall
Hi Julien,
Quoted 60 lines
Hi Dmytro,
Quoted 57 lines
Quoted 57 lines
Quoted 32 lines
Quoted 32 lines
Hi,
Quoted 27 lines · Volodymyr Babchuk
Quoted 13 lines · Julien Grall
Quoted 1 line · Volodymyr Babchuk
Hi Julien,
Hi Volodymyr,
Quoted 8 lines
Quoted 6 lines · Julien Grall
Hi,
Quoted 2 lines · On 16/06/2022 14:55, [email protected] wrote:
From: Dmytro Semenets <[email protected]> According to PSCI specification ARM TF can return DENIED on CPU OFF.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
I am confused. The spec is talking about Trusted OS and not firmware. The docummentation is also not specific to ARM Trusted Firmware. So did you mean "Trusted OS"?
It should be "firmware", I believe.
Hmmm... I couldn't find a reference in the spec suggesting that
CPU_OFFcould return DENIED because of the firmware. Do you have a pointer to the spec?Ah, looks like we are talking about different things. Indeed,
CPU_OFFcan return DENIED only because of Trusted OS. But entity that *returns* the error to a caller is a firmware.Quoted 6 lines
Quoted 2 lines
Right, the interesting part is *why* DENIED is returned not *who* returns it.
ARM TF returns DENIED *only* for the platform I have. We have a dissonance between spec and xen implementation because DENIED returned by
ARM TF or Thrusted OS or whatever is not a reason for panic.I agree that's not a reason for panic. However, knowing the reason does help to figure out the correct approach.
For instance, one could have suggest to migrate the trusted OS to another pCPU. But this would not have worked for you because the DENIED is not about that.
Quoted 4 lines
And we
have issues with this.
Ifmachine_restart()behaviour is more or less correct (sometimes reports about panic but restarts the machine)Right...
Quoted 1 line
but
machine_halt()doesn't work at al... this should also be the case here because
machine_halt()could also be called from cpu0. So I am a bit confused why you are saying it never works.
Quoted 8 lines
Quoted 5 lines
If
machine_halt()called on a CPU other than CPU0 it caused panic and reboot. If it called on a CPU0 it also caused panic but after system power off and reboot
is not issued. In this state you can still call the xen console. But you can't reboot the system.I am lost. In a previous e-mail you said that PSCI
CPU_OFFwould return DENIED on CPU0. IOW, I understood that for other CPUs, it would succeed.
I'm sorry I confused You.
Yes it causes panic and prints it will be rebooted but actual reboot doesn't happen.
Quoted 19 lines
But here, you are tell me the opposite:
"If it called on a CPU0 it also caused panic but after system power off
and reboot".If
machine_halt()is called from CPU0, thenCPU_OFFshould not be called on CPU0. So where is that panic coming from?Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
Transit execution to CPU0 for my understanding is a workaround and this approach will fix
machine_restart()function but will not fixmachine_halt().I would say it is a more specific case of what the spec suggests (see below). But it should fix both
machine_restart()andmachine_halt()because the last CPU running will be CPU0. So Xen would call SYSTEM_* rather thanCPU_OF. So I don't understand why you think it will fix one but not the other.
Quoted 5 lines
Quoted 2 lines
Looks like this is specific for my HW case.
SYSTEM_OFFdoesn't stop the whole system.Hmmm... All the other CPUs should be off (or spinning with interrupt disabled), so are you saying that
SYSTEM_OFFreturn?
Yes. SYSTEM_OFF returns on my HW. This is reason when CPU_OFF for CPU0 happens.
Quoted 5 lines
Cheers,
--
Julien Grall
Hi,
Quoted 10 lines · Dmytro Semenets
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
but
machine_halt()doesn't work at al... this should also be the case here because
machine_halt()could also be called from cpu0. So I am a bit confused why you are saying it never works.If
machine_halt()called on a CPU other than CPU0 it caused panic and reboot. If it called on a CPU0 it also caused panic but after system power off and reboot
is not issued. In this state you can still call the xen console. But you can't reboot the system.
Quoted 5 lines
Quoted 2 lines
I am lost. In a previous e-mail you said that PSCI
CPU_OFFwould return DENIED on CPU0. IOW, I understood that for other CPUs, it would succeed.I'm sorry I confused You.
Yes it causes panic and prints it will be rebooted but actual reboot doesn't happen.
Ok. That's most likely because of the call to smp_call_function() in machine_restart(). It is using cpu_online_map to decide which CPUs to send the IPI.
This will block because some of the CPUs are already off. So they will never acknowledge the IPI.
Quoted 20 lines
Quoted 20 lines
But here, you are tell me the opposite:
"If it called on a CPU0 it also caused panic but after system power off
and reboot".If
machine_halt()is called from CPU0, thenCPU_OFFshould not be called on CPU0. So where is that panic coming from?Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
Transit execution to CPU0 for my understanding is a workaround and this approach will fix
machine_restart()function but will not fixmachine_halt().I would say it is a more specific case of what the spec suggests (see below). But it should fix both
machine_restart()andmachine_halt()because the last CPU running will be CPU0. So Xen would call SYSTEM_* rather thanCPU_OF. So I don't understand why you think it will fix one but not the other.
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Looks like this is specific for my HW case.
SYSTEM_OFFdoesn't stop the whole system.Hmmm... All the other CPUs should be off (or spinning with interrupt disabled), so are you saying that
SYSTEM_OFFreturn?Yes.
SYSTEM_OFFreturns on my HW.
Hmmm... This is not compliant with the specification. Could you check why PSCI SYSTEM_OFF is returning?
Quoted 1 line
This is reason when
CPU_OFFfor CPU0 happens.
Right, machine_halt() will call halt_this_cpu() that in turn will call PSCI CPU_OFF.
If you modify halt_this_cpu() to avoid calling PSCI CPU_OFF (as I suggested before) then the panic() will never happen. Instead, the CPU will execute "wfi" in a loop with interrupt disabled.
To summarize there are two parts to resolve:
1) Understand why PSCI SYSTEM_OFF returns on your platform
2) Modify stop_cpu() to not call PSCI CPU_OFF
Cheers,
Signature · 1 line
Julien Grall