Skip to contentSkip to conversation

i2c: mv64xxx: Fix random system lock caused by runtime PM

PATCH Applied Marek Behún 4 participants 7 messages Apr 8, 2021 – Apr 15, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 i2c: mv64xxx: Fix random system lock caused by runtime PM

I noticed a weird bug with this driver on Marvell CN9130 Customer Reference Board.

Sometime after boot, the system locks with the following message:

 [104.071363] i2c i2c-0: mv64xxx: I2C bus locked, block: 1, time_left: 0

The system does not respond afterwards, only warns about RCU stalls.

This first appeared with commit e5c02cf54154 ("i2c: mv64xxx: Add runtime PM support").

With further experimentation I discovered that adding a delay into mv64xxx_i2c_hw_init() fixes this issue. This function is called before every xfer, due to how runtime PM works in this driver. It seems that in order to work correctly, a delay is needed after the bus is reset in this function.

Since there already is a known erratum with this controller needing a delay, I assume that this is just another place this needs to be applied. Therefore I apply the delay only if errata_delay is true.

---
 drivers/i2c/busses/i2c-mv64xxx.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/i2c/busses/i2c-mv64xxx.c +4 -0
index c590d36b5fd1..5c8e94b6cdb5 100644
@@ -221,6 +221,10 @@ mv64xxx_i2c_hw_init(struct mv64xxx_i2c_data *drv_data)
 	writel(0, drv_data->reg_base + drv_data->reg_offsets.ext_addr);
 	writel(MV64XXX_I2C_REG_CONTROL_TWSIEN | MV64XXX_I2C_REG_CONTROL_STOP,
 		drv_data->reg_base + drv_data->reg_offsets.control);
+
+	if (drv_data->errata_delay)
+		udelay(5);
+
 	drv_data->state = MV64XXX_I2C_STATE_IDLE;
 }
 
Signature · 1 line
2.26.2
#1 lore raw
Quoted 2 lines · On Thu, 8 Apr 2021 04:00:00 +0200 Marek Behún <[email protected]> wrote:

e5c02cf54154 ("i2c: mv64xxx: Add runtime
PM support").

This commit should also contain a Fixes tag:

Fixes e5c02cf54154 ("i2c: mv64xxx: Add runtime PM support").
#2 lore raw
In reply to Marek Behún
Quoted 17 lines · Marek Behún

I noticed a weird bug with this driver on Marvell CN9130 Customer Reference Board.

Sometime after boot, the system locks with the following message:

 [104.071363] i2c i2c-0: mv64xxx: I2C bus locked, block: 1, time_left: 0

The system does not respond afterwards, only warns about RCU stalls.

This first appeared with commit e5c02cf54154 ("i2c: mv64xxx: Add runtime PM support").

With further experimentation I discovered that adding a delay into mv64xxx_i2c_hw_init() fixes this issue. This function is called before every xfer, due to how runtime PM works in this driver. It seems that in order to work correctly, a delay is needed after the bus is reset in this function.

Quoted 5 lines

Since there already is a known erratum with this controller needing a delay, I assume that this is just another place this needs to be applied. Therefore I apply the delay only if errata_delay is true.

Gregory? Looks reasonable to me and if so, we should have this in 5.12 already. Comments from others are welcome, too, of course.

Quoted 22 lines
---
 drivers/i2c/busses/i2c-mv64xxx.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/i2c/busses/i2c-mv64xxx.c +4 -0
index c590d36b5fd1..5c8e94b6cdb5 100644
@@ -221,6 +221,10 @@ mv64xxx_i2c_hw_init(struct mv64xxx_i2c_data *drv_data)
 	writel(0, drv_data->reg_base + drv_data->reg_offsets.ext_addr);
 	writel(MV64XXX_I2C_REG_CONTROL_TWSIEN | MV64XXX_I2C_REG_CONTROL_STOP,
 		drv_data->reg_base + drv_data->reg_offsets.control);
+
+	if (drv_data->errata_delay)
+		udelay(5);
+
 	drv_data->state = MV64XXX_I2C_STATE_IDLE;
 }
 
Signature · 1 line
2.26.2
maintainer Acked-by
#3 lore raw
Quoted 9 lines · Marek Behún
Quoted 8 lines

I noticed a weird bug with this driver on Marvell CN9130 Customer Reference Board.

Sometime after boot, the system locks with the following message:

 [104.071363] i2c i2c-0: mv64xxx: I2C bus locked, block: 1, time_left: 0

The system does not respond afterwards, only warns about RCU stalls.

Quoted 8 lines
Quoted 8 lines

This first appeared with commit e5c02cf54154 ("i2c: mv64xxx: Add runtime PM support").

With further experimentation I discovered that adding a delay into mv64xxx_i2c_hw_init() fixes this issue. This function is called before every xfer, due to how runtime PM works in this driver. It seems that in order to work correctly, a delay is needed after the bus is reset in this function.

Marek,

As you mentioned it was related to reset and the issue occurred with the support of runtime PM. Did you try to add the delay only in the function mv64xxx_i2c_runtime_resume(), just after the mv64xxx_i2c_hw_init() call ?

Quoted 9 lines
Quoted 6 lines

Since there already is a known erratum with this controller needing a delay, I assume that this is just another place this needs to be applied. Therefore I apply the delay only if errata_delay is true.

Gregory? Looks reasonable to me and if so, we should have this in 5.12 already. Comments from others are welcome, too, of course.

Hello Wolfram,

I don't have this specific platform. However, as you said it looks reasonable and as it fixes an issue. And even if I had a pending question, it is just an optimisation so you can add my

Gregory

Quoted 23 lines
Quoted 23 lines
---
 drivers/i2c/busses/i2c-mv64xxx.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/i2c/busses/i2c-mv64xxx.c +4 -0
index c590d36b5fd1..5c8e94b6cdb5 100644
@@ -221,6 +221,10 @@ mv64xxx_i2c_hw_init(struct mv64xxx_i2c_data *drv_data)
 	writel(0, drv_data->reg_base + drv_data->reg_offsets.ext_addr);
 	writel(MV64XXX_I2C_REG_CONTROL_TWSIEN | MV64XXX_I2C_REG_CONTROL_STOP,
 		drv_data->reg_base + drv_data->reg_offsets.control);
+
+	if (drv_data->errata_delay)
+		udelay(5);
+
 	drv_data->state = MV64XXX_I2C_STATE_IDLE;
 }
 
Signature · 1 line
2.26.2
Signature · 3 lines
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com
#4 lore raw
Quoted 18 lines · On Wed, 14 Apr 2021 15:28:18 +0200 Gregory CLEMENT…
Quoted 9 lines · Marek Behún
Quoted 8 lines

I noticed a weird bug with this driver on Marvell CN9130 Customer Reference Board.

Sometime after boot, the system locks with the following message:

 [104.071363] i2c i2c-0: mv64xxx: I2C bus locked, block: 1, time_left: 0

The system does not respond afterwards, only warns about RCU stalls.

Quoted 8 lines
Quoted 8 lines

This first appeared with commit e5c02cf54154 ("i2c: mv64xxx: Add runtime PM support").

With further experimentation I discovered that adding a delay into mv64xxx_i2c_hw_init() fixes this issue. This function is called before every xfer, due to how runtime PM works in this driver. It seems that in order to work correctly, a delay is needed after the bus is reset in this function.

Quoted 6 lines

Marek,

As you mentioned it was related to reset and the issue occurred with the support of runtime PM. Did you try to add the delay only in the function mv64xxx_i2c_runtime_resume(), just after the mv64xxx_i2c_hw_init() call ?

I did indeed discover this when I added this delay into the resume function. In fact I discovered this when I added printf()s into suspend and resume when debugging. The problem disappeared with these printf()s (UART is slow so printf() counted as the necessary delay it seems).

I then moved the delay into the hw_init() function because that is what made sense to me, that the delay is necessary after the reset, not only when resuming, but always. We just did not notice because a xfer was never done immediately after reset before the PM patch. (But maybe I am wrong, maybe it is not needed in the reset. It just makes the most sense to me...)

Marek

Reviewed-by
#5 lore raw
In reply to Wolfram Sang
Quoted 18 lines · Wolfram Sang
Quoted 17 lines · Marek Behún

I noticed a weird bug with this driver on Marvell CN9130 Customer Reference Board.

Sometime after boot, the system locks with the following message:

 [104.071363] i2c i2c-0: mv64xxx: I2C bus locked, block: 1, time_left: 0

The system does not respond afterwards, only warns about RCU stalls.

This first appeared with commit e5c02cf54154 ("i2c: mv64xxx: Add runtime PM support").

With further experimentation I discovered that adding a delay into mv64xxx_i2c_hw_init() fixes this issue. This function is called before every xfer, due to how runtime PM works in this driver. It seems that in order to work correctly, a delay is needed after the bus is reset in this function.

Quoted 8 lines
Quoted 5 lines

Since there already is a known erratum with this controller needing a delay, I assume that this is just another place this needs to be applied. Therefore I apply the delay only if errata_delay is true.

Gregory? Looks reasonable to me and if so, we should have this in 5.12 already. Comments from others are welcome, too, of course.

I also don't have an affected platform, so I did not notice this when adding runtime PM. The change makes sense to me as well.

Cheers,
Samuel

Quoted 22 lines
Quoted 22 lines
---
 drivers/i2c/busses/i2c-mv64xxx.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/i2c/busses/i2c-mv64xxx.c +4 -0
index c590d36b5fd1..5c8e94b6cdb5 100644
@@ -221,6 +221,10 @@ mv64xxx_i2c_hw_init(struct mv64xxx_i2c_data *drv_data)
 	writel(0, drv_data->reg_base + drv_data->reg_offsets.ext_addr);
 	writel(MV64XXX_I2C_REG_CONTROL_TWSIEN | MV64XXX_I2C_REG_CONTROL_STOP,
 		drv_data->reg_base + drv_data->reg_offsets.control);
+
+	if (drv_data->errata_delay)
+		udelay(5);
+
 	drv_data->state = MV64XXX_I2C_STATE_IDLE;
 }
 
Signature · 1 line
2.26.2
Applied
#6 lore raw
In reply to Marek Behún
Quoted 17 lines · Marek Behún

I noticed a weird bug with this driver on Marvell CN9130 Customer Reference Board.

Sometime after boot, the system locks with the following message:

 [104.071363] i2c i2c-0: mv64xxx: I2C bus locked, block: 1, time_left: 0

The system does not respond afterwards, only warns about RCU stalls.

This first appeared with commit e5c02cf54154 ("i2c: mv64xxx: Add runtime PM support").

With further experimentation I discovered that adding a delay into mv64xxx_i2c_hw_init() fixes this issue. This function is called before every xfer, due to how runtime PM works in this driver. It seems that in order to work correctly, a delay is needed after the bus is reset in this function.

Quoted 5 lines

Since there already is a known erratum with this controller needing a delay, I assume that this is just another place this needs to be applied. Therefore I apply the delay only if errata_delay is true.

Applied to for-current, thanks!