Fix RTC and NVRAM OPAL_BUSY loops
Subsystems (2)
Nicholas Piggin
reviewercover Fix RTC and NVRAM OPAL_BUSY loops
This is a couple of important fixes broken out of the series "first step of standardising OPAL_BUSY handling", that prevents the kernel from locking up if the NVRAM or RTC hardware does not respond.
Another one, the console driver, has a similar problem that has also been hit in testing, but that requires larger fixes to the opal console and hvc tty driver that won't make it for 4.17.
Thanks,
Nick
Nicholas Piggin (3):
powerpc/powernv: define a standard delay for OPAL_BUSY type retry
loops
powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops powerpc/powernv: Fix OPAL NVRAM driver OPAL_BUSY loops
arch/powerpc/include/asm/opal.h | 3 ++ arch/powerpc/platforms/powernv/opal-nvram.c | 7 +++- arch/powerpc/platforms/powernv/opal-rtc.c | 8 +++-- drivers/rtc/rtc-opal.c | 37 +++++++++++++-------- 4 files changed, 37 insertions(+), 18 deletions(-)
Signature · 1 line
2.17.0
Nicholas Piggin
reviewerPATCH 1/3 powerpc/powernv: define a standard delay for OPAL_BUSY type retry loops
This is the start of an effort to tidy up and standardise all the delays. Existing loops have a range of delay/sleep periods from 1ms to 20ms, and some have no delay. They all loop forever except rtc, which times out after 10 retries, and that uses 10ms delays. So use 10ms as our standard delay. The OPAL maintainer agrees 10ms is a reasonable starting point.
The idea is to use the same recipe everywhere, once this is proven to work then it will be documented as an OPAL API standard. Then both firmware and OS can agree, and if a particular call needs something else, then that can be documented with reasoning.
This is not the end-all of this effort, it's just a relatively easy change that fixes some existing high latency delays. There should be provision for standardising timeouts and/or interruptible loops where possible, so non-fatal firmware errors don't cause hangs.
--- arch/powerpc/include/asm/opal.h | 3 +++ 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/opal.h +3 -0
@@ -21,6 +21,9 @@/* We calculate number of sg entries based on PAGE_SIZE */ #define SG_ENTRIES_PER_NODE ((PAGE_SIZE - 16) / sizeof(struct opal_sg_entry)) +/* Default time to sleep or delay between OPAL_BUSY/OPAL_BUSY_EVENT loops */ +#define OPAL_BUSY_DELAY_MS 10 +/* /sys/firmware/opal */ extern struct kobject *opal_kobj;
Signature · 1 line
2.17.0
Michael Ellerman
AppliedRe: [1/3] powerpc/powernv: define a standard delay for OPAL_BUSY type retry loops
Quoted 12 lines · Nicholas Piggin
This is the start of an effort to tidy up and standardise all the delays. Existing loops have a range of delay/sleep periods from 1ms to 20ms, and some have no delay. They all loop forever except rtc, which times out after 10 retries, and that uses 10ms delays. So use 10ms as our standard delay. The OPAL maintainer agrees 10ms is a reasonable starting point.
The idea is to use the same recipe everywhere, once this is proven to work then it will be documented as an OPAL API standard. Then both firmware and OS can agree, and if a particular call needs something else, then that can be documented with reasoning.
Quoted 6 lines
This is not the end-all of this effort, it's just a relatively easy change that fixes some existing high latency delays. There should be provision for standardising timeouts and/or interruptible loops where possible, so non-fatal firmware errors don't cause hangs.
Signed-off-by Nicholas Piggin <[email protected]>
Applied to powerpc fixes, thanks.
https://git.kernel.org/powerpc/c/34dd25de9fe3f60bfdb31b473bf04b
cheers
Nicholas Piggin
reviewerPATCH 2/3 powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops
In reply to Nicholas PigginThe OPAL RTC driver does not sleep in case it gets OPAL_BUSY or OPAL_BUSY_EVENT from firmware, which causes large scheduling latencies, up to 50 seconds have been observed here when RTC stops responding (BMC reboot can do it).
Fix this by converting it to the standard form OPAL_BUSY loop that sleeps.
Fixes 628daa8d5abfd ("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks"
--- arch/powerpc/platforms/powernv/opal-rtc.c | 8 +++-- drivers/rtc/rtc-opal.c | 37 ++++++++++++++--------- 2 files changed, 28 insertions(+), 17 deletions(-)
arch/powerpc/platforms/powernv/opal-rtc.c +5 -3
@@ -48,10 +48,12 @@ unsigned long __init opal_get_boot_time(void)while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ) { rc =opal_rtc_read (&__y_m_d, &__h_m_s_ms); -if (rc ==OPAL_BUSY_EVENT ) +if (rc ==OPAL_BUSY_EVENT ) { +mdelay (OPAL_BUSY_DELAY_MS );opal_poll_events (NULL ); -else if (rc ==OPAL_BUSY ) -mdelay (10 ); + }else if (rc ==OPAL_BUSY ) { +mdelay (OPAL_BUSY_DELAY_MS ); + } }if (rc !=OPAL_SUCCESS )return 0 ;
drivers/rtc/rtc-opal.c +23 -14
@@ -57,7 +57,7 @@ static void tm_to_opal(struct rtc_time *tm, u32 *y_m_d, u64 *h_m_s_ms)static int opal_get_rtc_time (struct device *dev,struct rtc_time *tm) { -long rc =OPAL_BUSY ; +s64 rc =OPAL_BUSY ;int retries =10 ;u32 y_m_d;u64 h_m_s_ms;
@@ -66,13 +66,17 @@ static int opal_get_rtc_time(struct device *dev, struct rtc_time *tm)while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ) { rc =opal_rtc_read (&__y_m_d, &__h_m_s_ms); -if (rc ==OPAL_BUSY_EVENT ) +if (rc ==OPAL_BUSY_EVENT ) { +msleep (OPAL_BUSY_DELAY_MS );opal_poll_events (NULL ); -else if (retries-- && (rc ==OPAL_HARDWARE - || rc ==OPAL_INTERNAL_ERROR )) -msleep (10 ); -else if (rc !=OPAL_BUSY && rc !=OPAL_BUSY_EVENT ) -break ; + }else if (rc ==OPAL_BUSY ) { +msleep (OPAL_BUSY_DELAY_MS ); + }else if (rc ==OPAL_HARDWARE || rc ==OPAL_INTERNAL_ERROR ) { +if (retries--) { +msleep (10 );/* Wait 10ms before retry */ + rc =OPAL_BUSY ;/* go around again */ + } + } }if (rc !=OPAL_SUCCESS )
@@ -87,21 +91,26 @@ static int opal_get_rtc_time(struct device *dev, struct rtc_time *tm)static int opal_set_rtc_time (struct device *dev,struct rtc_time *tm) { -long rc =OPAL_BUSY ; +s64 rc =OPAL_BUSY ;int retries =10 ;u32 y_m_d =0 ;u64 h_m_s_ms =0 ;tm_to_opal (tm, &y_m_d, &h_m_s_ms); +while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ) { rc =opal_rtc_write (y_m_d, h_m_s_ms); -if (rc ==OPAL_BUSY_EVENT ) +if (rc ==OPAL_BUSY_EVENT ) { +msleep (OPAL_BUSY_DELAY_MS );opal_poll_events (NULL ); -else if (retries-- && (rc ==OPAL_HARDWARE - || rc ==OPAL_INTERNAL_ERROR )) -msleep (10 ); -else if (rc !=OPAL_BUSY && rc !=OPAL_BUSY_EVENT ) -break ; + }else if (rc ==OPAL_BUSY ) { +msleep (OPAL_BUSY_DELAY_MS ); + }else if (rc ==OPAL_HARDWARE || rc ==OPAL_INTERNAL_ERROR ) { +if (retries--) { +msleep (10 );/* Wait 10ms before retry */ + rc =OPAL_BUSY ;/* go around again */ + } + } }return rc ==OPAL_SUCCESS ?0 : -EIO ;
Signature · 1 line
2.17.0
Alexandre Belloni
maintainerRe: [PATCH 2/3] powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops
Hi Nicholas,
I would greatly appreciate a changelog and at least the cover letter because it is difficult to grasp how this relates to the previous patches you sent to the RTC mailing list.
Quoted 9 lines · Nicholas Piggin
The OPAL RTC driver does not sleep in case it gets
OPAL_BUSYorOPAL_BUSY_EVENTfrom firmware, which causes large scheduling latencies, up to 50 seconds have been observed here when RTC stops responding (BMC reboot can do it).Fix this by converting it to the standard form
OPAL_BUSYloop that sleeps.Fixes
628daa8d5abfd("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks"
Quoted 8 lines
Cc Benjamin Herrenschmidt <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/platforms/powernv/opal-rtc.c | 8 +++-- drivers/rtc/rtc-opal.c | 37 ++++++++++++++---------From what I understand, the changes in those files are fairly
independent, they should probably be separated to ease merging.
Quoted 96 lines
2 files changed, 28 insertions(+), 17 deletions(-)arch/powerpc/platforms/powernv/opal-rtc.c +5
-3@@ -48,10 +48,12 @@ unsigned long __init opal_get_boot_time(void)while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ) { rc =opal_rtc_read (&__y_m_d, &__h_m_s_ms); -if (rc ==OPAL_BUSY_EVENT ) +if (rc ==OPAL_BUSY_EVENT ) { +mdelay (OPAL_BUSY_DELAY_MS );opal_poll_events (NULL ); -else if (rc ==OPAL_BUSY ) -mdelay (10 ); + }else if (rc ==OPAL_BUSY ) { +mdelay (OPAL_BUSY_DELAY_MS ); + } }if (rc !=OPAL_SUCCESS )return 0 ;drivers/rtc/rtc-opal.c +23
-14@@ -57,7 +57,7 @@ static void tm_to_opal(struct rtc_time *tm, u32 *y_m_d, u64 *h_m_s_ms)static int opal_get_rtc_time (struct device *dev,struct rtc_time *tm) { -long rc =OPAL_BUSY ; +s64 rc =OPAL_BUSY ;int retries =10 ;u32 y_m_d;u64 h_m_s_ms;@@ -66,13 +66,17 @@ static int opal_get_rtc_time(struct device *dev, struct rtc_time *tm)while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ) { rc =opal_rtc_read (&__y_m_d, &__h_m_s_ms); -if (rc ==OPAL_BUSY_EVENT ) +if (rc ==OPAL_BUSY_EVENT ) { +msleep (OPAL_BUSY_DELAY_MS );opal_poll_events (NULL ); -else if (retries-- && (rc ==OPAL_HARDWARE - || rc ==OPAL_INTERNAL_ERROR )) -msleep (10 ); -else if (rc !=OPAL_BUSY && rc !=OPAL_BUSY_EVENT ) -break ; + }else if (rc ==OPAL_BUSY ) { +msleep (OPAL_BUSY_DELAY_MS ); + }else if (rc ==OPAL_HARDWARE || rc ==OPAL_INTERNAL_ERROR ) { +if (retries--) { +msleep (10 );/* Wait 10ms before retry */ + rc =OPAL_BUSY ;/* go around again */ + } + } }if (rc !=OPAL_SUCCESS )@@ -87,21 +91,26 @@ static int opal_get_rtc_time(struct device *dev, struct rtc_time *tm)static int opal_set_rtc_time (struct device *dev,struct rtc_time *tm) { -long rc =OPAL_BUSY ; +s64 rc =OPAL_BUSY ;int retries =10 ;u32 y_m_d =0 ;u64 h_m_s_ms =0 ;tm_to_opal (tm, &y_m_d, &h_m_s_ms); +while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ) { rc =opal_rtc_write (y_m_d, h_m_s_ms); -if (rc ==OPAL_BUSY_EVENT ) +if (rc ==OPAL_BUSY_EVENT ) { +msleep (OPAL_BUSY_DELAY_MS );opal_poll_events (NULL ); -else if (retries-- && (rc ==OPAL_HARDWARE - || rc ==OPAL_INTERNAL_ERROR )) -msleep (10 ); -else if (rc !=OPAL_BUSY && rc !=OPAL_BUSY_EVENT ) -break ; + }else if (rc ==OPAL_BUSY ) { +msleep (OPAL_BUSY_DELAY_MS ); + }else if (rc ==OPAL_HARDWARE || rc ==OPAL_INTERNAL_ERROR ) { +if (retries--) { +msleep (10 );/* Wait 10ms before retry */ + rc =OPAL_BUSY ;/* go around again */ + } + } }return rc ==OPAL_SUCCESS ?0 : -EIO ;Signature · 1 line
2.17.0
Signature · 3 lines
Alexandre Belloni, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com
Nicholas Piggin
reviewerRe: [PATCH 2/3] powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops
Quoted 5 lines · On Tue, 10 Apr 2018 14:07:28 +0200 Alexandre Belloni…
Hi Nicholas,
I would greatly appreciate a changelog and at least the cover letter because it is difficult to grasp how this relates to the previous patches you sent to the RTC mailing list.
Yes good point. Basically this change is "standalone" except using OPAL_BUSY_DELAY_MS define from patch 1. That patch has a lot of comments about firmware delays I did not think would be too interesting.
Basically we're adding msleep(10) here, because the firmware can repeatedly return OPAL_BUSY for long periods, so we want to context switch and respond to interrupts.
Quoted 18 lines · Nicholas Piggin
Quoted 8 lines
The OPAL RTC driver does not sleep in case it gets
OPAL_BUSYorOPAL_BUSY_EVENTfrom firmware, which causes large scheduling latencies, up to 50 seconds have been observed here when RTC stops responding (BMC reboot can do it).Fix this by converting it to the standard form
OPAL_BUSYloop that sleeps.Quoted 7 lines
Fixes ("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks"
Cc Benjamin Herrenschmidt <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/platforms/powernv/opal-rtc.c | 8 +++-- drivers/rtc/rtc-opal.c | 37 ++++++++++++++---------
Quoted 2 lines
From what I understand, the changes in those files are fairly independent, they should probably be separated to ease merging.
I'm happy to do that. It's using the same firmware call, so I thought a single patch would be fine. But I guess the boot call can be dropped from this patch because it does not not solve the problem described in the changelog.
Would you be happy for the driver change to be merged via the powerpc tree? The code being fixed here came from the same original patch as a similar issue being fixed in the OPAL NVRAM driver so it might be easier that way.
Thanks,
Nick
Re: [PATCH 2/3] powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops
Quoted 18 lines · Nicholas Piggin
Quoted 10 lines · On Tue, 10 Apr 2018 14:07:28 +0200 Alexandre Belloni…
Quoted 7 lines
Fixes ("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks"
Cc Benjamin Herrenschmidt <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/platforms/powernv/opal-rtc.c | 8 +++-- drivers/rtc/rtc-opal.c | 37 ++++++++++++++---------From what I understand, the changes in those files are fairly independent, they should probably be separated to ease merging.
I'm happy to do that. It's using the same firmware call, so I thought a single patch would be fine. But I guess the boot call can be dropped from this patch because it does not not solve the problem described in the changelog.
Quoted 5 lines
Would you be happy for the driver change to be merged via the powerpc tree? The code being fixed here came from the same original patch as a similar issue being fixed in the OPAL NVRAM driver so it might be easier that way.
Ok then, just add my
and let it go through the powerpc tree.
Signature · 3 lines
Alexandre Belloni, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com
Michael Ellerman
reviewerRe: [PATCH 2/3] powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops
Quoted 24 lines · Alexandre Belloni
Quoted 18 lines · Nicholas Piggin
Quoted 10 lines · On Tue, 10 Apr 2018 14:07:28 +0200 Alexandre Belloni…
Quoted 7 lines
Fixes ("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks"
Cc Benjamin Herrenschmidt <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/platforms/powernv/opal-rtc.c | 8 +++-- drivers/rtc/rtc-opal.c | 37 ++++++++++++++---------From what I understand, the changes in those files are fairly independent, they should probably be separated to ease merging.
I'm happy to do that. It's using the same firmware call, so I thought a single patch would be fine. But I guess the boot call can be dropped from this patch because it does not not solve the problem described in the changelog.
Quoted 4 lines
Would you be happy for the driver change to be merged via the powerpc tree? The code being fixed here came from the same original patch as a similar issue being fixed in the OPAL NVRAM driver so it might be easier that way.
Quoted 5 lines
Ok then, just add my
and let it go through the powerpc tree.
Thanks.
It's still mostly an rtc patch by lines changed, so I changed the subject to:
rtc: opal: Fix OPAL RTC driver OPAL_BUSY loops
cheers
Alexandre Belloni
maintainerRe: [PATCH 2/3] powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops
Quoted 29 lines · Michael Ellerman
Quoted 28 lines · Alexandre Belloni
Quoted 18 lines · Nicholas Piggin
Quoted 10 lines · On Tue, 10 Apr 2018 14:07:28 +0200 Alexandre Belloni…
Quoted 7 lines
Fixes ("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks"
Cc Benjamin Herrenschmidt <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/platforms/powernv/opal-rtc.c | 8 +++-- drivers/rtc/rtc-opal.c | 37 ++++++++++++++---------From what I understand, the changes in those files are fairly independent, they should probably be separated to ease merging.
I'm happy to do that. It's using the same firmware call, so I thought a single patch would be fine. But I guess the boot call can be dropped from this patch because it does not not solve the problem described in the changelog.
Quoted 4 lines
Would you be happy for the driver change to be merged via the powerpc tree? The code being fixed here came from the same original patch as a similar issue being fixed in the OPAL NVRAM driver so it might be easier that way.
Ok then, just add my
Quoted 9 lines
Quoted 1 line
and let it go through the powerpc tree.
Thanks.
It's still mostly an rtc patch by lines changed, so I changed the subject to:
rtc: opal: Fix OPAL RTC driver
OPAL_BUSYloops
Great, thanks!
Signature · 3 lines
Alexandre Belloni, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com
Michael Ellerman
AppliedRe: [2/3] powerpc/powernv: Fix OPAL RTC driver OPAL_BUSY loops
In reply to Nicholas PigginQuoted 5 lines · Nicholas Piggin
The OPAL RTC driver does not sleep in case it gets
OPAL_BUSYorOPAL_BUSY_EVENTfrom firmware, which causes large scheduling latencies, up to 50 seconds have been observed here when RTC stops responding (BMC reboot can do it).
Quoted 8 lines
Fix this by converting it to the standard form
OPAL_BUSYloop that sleeps.Fixes
628daa8d5abfd("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks"Cc Benjamin Herrenschmidt <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>
Applied to powerpc fixes, thanks.
https://git.kernel.org/powerpc/c/682e6b4da5cbe8e9a53f979a58c2a9
cheers
Nicholas Piggin
reviewerPATCH 3/3 powerpc/powernv: Fix OPAL NVRAM driver OPAL_BUSY loops
In reply to Nicholas PigginThe OPAL NVRAM driver does not sleep in case it gets OPAL_BUSY or OPAL_BUSY_EVENT from firmware, which causes large scheduling latencies, and various lockup errors to trigger (again, BMC reboot can cause it).
Fix this by converting it to the standard form OPAL_BUSY loop that sleeps.
--- arch/powerpc/platforms/powernv/opal-nvram.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/powerpc/platforms/powernv/opal-nvram.c +6 -1
@@ -11,6 +11,7 @@#define DEBUG +#include <linux/delay.h> #include <linux/kernel.h> #include <linux/init.h> #include <linux/of.h>
@@ -56,8 +57,12 @@ static ssize_t opal_nvram_write(char *buf, size_t count, loff_t *index)while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ) { rc =opal_write_nvram (__pa (buf), count, off); -if (rc ==OPAL_BUSY_EVENT ) +if (rc ==OPAL_BUSY_EVENT ) { +msleep (OPAL_BUSY_DELAY_MS );opal_poll_events (NULL ); + }else if (rc ==OPAL_BUSY ) { +msleep (OPAL_BUSY_DELAY_MS ); + } }if (rc)
Signature · 1 line
2.17.0
Michael Ellerman
AppliedRe: [3/3] powerpc/powernv: Fix OPAL NVRAM driver OPAL_BUSY loops
Quoted 11 lines · Nicholas Piggin
The OPAL NVRAM driver does not sleep in case it gets
OPAL_BUSYorOPAL_BUSY_EVENTfrom firmware, which causes large scheduling latencies, and various lockup errors to trigger (again, BMC reboot can cause it).Fix this by converting it to the standard form
OPAL_BUSYloop that sleeps.Fixes 628daa8d5abfd ("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks")Cc Benjamin Herrenschmidt <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>
Applied to powerpc fixes, thanks.
https://git.kernel.org/powerpc/c/3b8070335f751aac9f1526ae2e012e
cheers