powerpc/rtas: retry when cpu offline races with suspend/migration
PATCH 1/1 powerpc/rtas: retry when cpu offline races with suspend/migration
The protocol for suspending or migrating an LPAR requires all present processor threads to enter H_JOIN. So if we have threads offline, we have to temporarily bring them up. This can race with administrator actions such as SMT state changes. As of dfd718a2ed1f ("powerpc/rtas:
Fix a potential race between CPU-Offline & Migration"), rtas_ibm_suspend_me() accounts for this, but errors out with -EBUSY for what almost certainly is a transient condition in any reasonable scenario.
Callers of rtas_ibm_suspend_me() already retry when -EAGAIN is returned, and it is typical during a migration for that to happen repeatedly for several minutes polling the H_VASI_STATE hcall result before proceeding to the next stage.
So return -EAGAIN instead of -EBUSY when this race is encountered. Additionally: logging this event is still appropriate but use pr_info instead of pr_err; and remove use of unlikely() while here as this is not a hot path at all.
--- arch/powerpc/kernel/rtas.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/powerpc/kernel/rtas.c +3 -4
@@ -984,10 +984,9 @@ int rtas_ibm_suspend_me(u64 handle)cpu_hotplug_disable ();/* Check if we raced with a CPU-Offline Operation */ -if (unlikely (!cpumask_equal (cpu_present_mask, cpu_online_mask))) { -pr_err ("%s: Raced against a concurrent CPU-Offline\n" , - __func__); -atomic_set (&data.error, -EBUSY ); +if (!cpumask_equal (cpu_present_mask, cpu_online_mask)) { +pr_info ("%s: Raced against a concurrent CPU-Offline\n" , __func__); +atomic_set (&data.error, -EAGAIN ); goto out_hotplug_enable; }
Signature · 1 line
2.20.1
Quoted 14 lines · Nathan Lynch
The protocol for suspending or migrating an LPAR requires all present processor threads to enter
H_JOIN. So if we have threads offline, we have to temporarily bring them up. This can race with administrator actions such as SMT state changes. As ofdfd718a2ed1f("powerpc/rtas:
Fix a potential race between CPU-Offline & Migration"),rtas_ibm_suspend_me()accounts for this, but errors out with-EBUSYfor what almost certainly is a transient condition in any reasonable scenario.Callers of
rtas_ibm_suspend_me()already retry when-EAGAINis returned, and it is typical during a migration for that to happen repeatedly for several minutes polling theH_VASI_STATEhcall result before proceeding to the next stage.
Quoted 5 lines
So return
-EAGAINinstead of-EBUSYwhen this race is encountered. Additionally: logging this event is still appropriate but usepr_infoinstead ofpr_err; and remove use ofunlikely()while here as this is not a hot path at all.
Looks good, since it's not a hot path anyway, so unlikely() should benefit from optimize compiler path, and should stay. No?
Mingming
Quoted 24 lines
Fixes dfd718a2ed1f ("powerpc/rtas: Fix a potential race betweenCPU-Offline & Migration")
Signed-off-by Nathan Lynch <[email protected]>--- arch/powerpc/kernel/rtas.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)arch/powerpc/kernel/rtas.c +3
-4@@ -984,10 +984,9 @@ int rtas_ibm_suspend_me(u64 handle)cpu_hotplug_disable ();/* Check if we raced with a CPU-Offline Operation */ -if (unlikely (!cpumask_equal (cpu_present_mask, cpu_online_mask))) { -pr_err ("%s: Raced against a concurrent CPU-Offline\n" , - __func__); -atomic_set (&data.error, -EBUSY ); +if (!cpumask_equal (cpu_present_mask, cpu_online_mask)) { +pr_info ("%s: Raced against a concurrent CPU-Offline\n" , __func__); +atomic_set (&data.error, -EAGAIN ); goto out_hotplug_enable; }
Hi Mingming,
Quoted 8 lines · mmc
Quoted 4 lines · Nathan Lynch
So return
-EAGAINinstead of-EBUSYwhen this race is encountered. Additionally: logging this event is still appropriate but usepr_infoinstead ofpr_err; and remove use ofunlikely()while here as this is not a hot path at all.Looks good, since it's not a hot path anyway, so
unlikely()should benefit from optimize compiler path, and should stay. No?
The latency of this path in rtas_ibm_suspend_me() in the best case is around 2-3 seconds.
So I think not -- this is such a heavyweight and relatively seldom-executed path that the unlikely() cannot yield any discernible performance benefit, and its presence imposes a readability cost.
There's some concern this could retry forever, resulting in live lock. Another approach would be to abandon the attempt and fail the LPM request. https://lists.ozlabs.org/pipermail/linuxppc-dev/2019-June/192531.html
Quoted 16 lines · Nathan Lynch
Hi Mingming,
Quoted 7 lines · mmc
Quoted 4 lines · Nathan Lynch
So return
-EAGAINinstead of-EBUSYwhen this race is encountered. Additionally: logging this event is still appropriate but usepr_infoinstead ofpr_err; and remove use ofunlikely()while here as this is not a hot path at all.Looks good, since it's not a hot path anyway, so
unlikely()should benefit from optimize compiler path, and should stay. No?The latency of this path in
rtas_ibm_suspend_me()in the best case is around 2-3 seconds.So I think not -- this is such a heavyweight and relatively seldom-executed path that the
unlikely()cannot yield any discernible performance benefit, and its presence imposes a readability cost.
Quoted 1 line · Juliet Kim
There's some concern this could retry forever, resulting in live lock.
First of all the system will make progress in other areas even if there are repeated retries; we're not indefinitely holding locks or anything like that.
Second, Linux checks the H_VASI_STATE result on every retry. If the platform wants to terminate the migration (say, if it imposes a timeout), Linux will abandon it when H_VASI_STATE fails to return H_VASI_SUSPENDING. And it seems incorrect to bail out before that happens, absent hard errors on the Linux side such as allocation failures.
Quoted 6 lines · Nathan Lynch
Quoted 1 line · Juliet Kim
There's some concern this could retry forever, resulting in live lock.
First of all the system will make progress in other areas even if there are repeated retries; we're not indefinitely holding locks or anything like that.
For instance, system admin runs a script that picks and offlines CPUs in a loop to keep a certain rate of onlined CPUs for energy saving. If LPM keeps putting CPUs back online, that would never finish, and would keepgenerating new offline requests
Quoted 6 lines
Second, Linux checks the
H_VASI_STATEresult on every retry. If the platform wants to terminate the migration (say, if it imposes a timeout), Linux will abandon it whenH_VASI_STATEfails to returnH_VASI_SUSPENDING. And it seems incorrect to bail out before that happens, absent hard errors on the Linux side such as allocation failures.
I confirmed with the PHYP and HMC folks that they wouldn't time out the LPM request including H_VASI_STATE, so if the LPM retries were unlucky enough to encounter repeated CPU offline attempts (maybe some customer code retrying that), then the retries could continue indefinitely, or until some manual intervention. And in the mean time, the LPM delay here would cause PHYP to block other operations.
Michael Ellerman
reviewerQuoted 19 lines · Juliet Kim
Quoted 6 lines · Nathan Lynch
Quoted 1 line · Juliet Kim
There's some concern this could retry forever, resulting in live lock.
First of all the system will make progress in other areas even if there are repeated retries; we're not indefinitely holding locks or anything like that.
For instance, system admin runs a script that picks and offlines CPUs in a loop to keep a certain rate of onlined CPUs for energy saving. If LPM keeps putting CPUs back online, that would never finish, and would keepgenerating new offline requests
Quoted 6 lines
Second, Linux checks the
H_VASI_STATEresult on every retry. If the platform wants to terminate the migration (say, if it imposes a timeout), Linux will abandon it whenH_VASI_STATEfails to returnH_VASI_SUSPENDING. And it seems incorrect to bail out before that happens, absent hard errors on the Linux side such as allocation failures.
Quoted 6 lines
I confirmed with the PHYP and HMC folks that they wouldn't time out the LPM request including
H_VASI_STATE, so if the LPM retries were unlucky enough to encounter repeated CPU offline attempts (maybe some customer code retrying that), then the retries could continue indefinitely, or until some manual intervention. And in the mean time, the LPM delay here would cause PHYP to block other operations.
That sounds like a PHYP bug to me.
cheers
Quoted 25 lines · Michael Ellerman
Quoted 18 lines · Juliet Kim
Quoted 6 lines · Nathan Lynch
Quoted 1 line · Juliet Kim
There's some concern this could retry forever, resulting in live lock.
First of all the system will make progress in other areas even if there are repeated retries; we're not indefinitely holding locks or anything like that.
For instance, system admin runs a script that picks and offlines CPUs in a loop to keep a certain rate of onlined CPUs for energy saving. If LPM keeps putting CPUs back online, that would never finish, and would keepgenerating new offline requests
Quoted 6 lines
Second, Linux checks the
H_VASI_STATEresult on every retry. If the platform wants to terminate the migration (say, if it imposes a timeout), Linux will abandon it whenH_VASI_STATEfails to returnH_VASI_SUSPENDING. And it seems incorrect to bail out before that happens, absent hard errors on the Linux side such as allocation failures.Quoted 6 lines
I confirmed with the PHYP and HMC folks that they wouldn't time out the LPM request including
H_VASI_STATE, so if the LPM retries were unlucky enough to encounter repeated CPU offline attempts (maybe some customer code retrying that), then the retries could continue indefinitely, or until some manual intervention. And in the mean time, the LPM delay here would cause PHYP to block other operations.
Quoted 3 lines
That sounds like a PHYP bug to me.
cheers
PHYP doesn’t time out because they have no idea how long it will take for OS to get to the point that it suspends. Other OS allows application to prepare for LPM. They cannot predict the length of time that is appropriate in all cases and in any case, it’s unlikely they’d make a change to that would come in time to help with the current issue.
Hi Michael,
Quoted 5 lines · Nathan Lynch
The protocol for suspending or migrating an LPAR requires all present processor threads to enter
H_JOIN. So if we have threads offline, we have to temporarily bring them up. This can race with administrator actions such as SMT state changes. As ofdfd718a2ed1f("powerpc/rtas:
Fix a potential race between CPU-Offline & Migration"),
snowpatch/checkpatch flagged an error in my commit message here:
ERROR:GIT_COMMIT_ID: Please use git commit description style 'commit <12+ chars of sha1> ("<title line>")' - ie: 'commit dfd718a2ed1f ("powerpc/rtas: Fix a potential race between CPU-Offline & Migration")'
I see this is in your next-test branch though. Should I fix the commit message and resend?
Michael Ellerman
reviewerQuoted 11 lines · Nathan Lynch
Hi Michael,
Quoted 5 lines · Nathan Lynch
The protocol for suspending or migrating an LPAR requires all present processor threads to enter
H_JOIN. So if we have threads offline, we have to temporarily bring them up. This can race with administrator actions such as SMT state changes. As ofdfd718a2ed1f("powerpc/rtas:
Fix a potential race between CPU-Offline & Migration"),snowpatch/checkpatch flagged an error in my commit message here:
Quoted 7 lines
ERROR:GIT_COMMIT_ID: Please use git commit description style 'commit <12+ chars of sha1> ("<title line>")' - ie: 'commit
dfd718a2ed1f("powerpc/rtas: Fix a potential race between CPU-Offline & Migration")'I see this is in your next-test branch though. Should I fix the commit message and resend?
No that's fine.
You have a Fixes tag which is formatted correctly and that's what matters IMO.
Ideally we could control that check to not complain about it being split across lines when there's a fixes tag as well.
cheers
Michael Ellerman
AppliedQuoted 14 lines · Nathan Lynch
The protocol for suspending or migrating an LPAR requires all present processor threads to enter
H_JOIN. So if we have threads offline, we have to temporarily bring them up. This can race with administrator actions such as SMT state changes. As ofdfd718a2ed1f("powerpc/rtas:
Fix a potential race between CPU-Offline & Migration"),rtas_ibm_suspend_me()accounts for this, but errors out with-EBUSYfor what almost certainly is a transient condition in any reasonable scenario.Callers of
rtas_ibm_suspend_me()already retry when-EAGAINis returned, and it is typical during a migration for that to happen repeatedly for several minutes polling theH_VASI_STATEhcall result before proceeding to the next stage.
Quoted 7 lines
So return
-EAGAINinstead of-EBUSYwhen this race is encountered. Additionally: logging this event is still appropriate but usepr_infoinstead ofpr_err; and remove use ofunlikely()while here as this is not a hot path at all.Fixes dfd718a2ed1f ("powerpc/rtas: Fix a potential race between CPU-Offline & Migration")Signed-off-by Nathan Lynch <[email protected]>
Applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/9fb603050ffd94f8127df99c699cca2f575eb6a0
cheers