Skip to contentSkip to conversation

Regulator regression in next-20180305

DISCUSSION Tony Lindgren 4 participants 14 messages Mar 5, 2018 – Mar 7, 2018 Voltage and current regulator framework

lore
Share & tools
lore
#0 lore raw

Hi Mark and Maciej,

Looks like with next-20180305 there's a regulator regression where mmc0 won't show any cards or produces errors:

mmcblk0: error -110 requesting status
mmc1: new high speed SDIO card at address 0001
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!
print_req_error: I/O error, dev mmcblk0, sector 0 Buffer I/O error on dev mmcblk0, logical block 0, async page read
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!

I bisected it down to the following commits:

broken 71c33626a802 ("regulator: core: Change voltage setting path")
fails to build a7348f502ab7 ("regulator: core: Add voltage balancing mechanism")
works cf6fc8064766 ("regulator: core: Resolve coupled regulators")

Any ideas?

Regards,

Tony

#1 lore raw

Hi Tony,

Quoted 14 lines · Tony Lindgren

Hi Mark and Maciej,

Looks like with next-20180305 there's a regulator regression where mmc0 won't show any cards or produces errors:

mmcblk0: error -110 requesting status
mmc1: new high speed SDIO card at address 0001
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!
print_req_error: I/O error, dev mmcblk0, sector 0 Buffer I/O error on dev mmcblk0, logical block 0, async page read
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!

Quoted 7 lines

I bisected it down to the following commits:

broken          71c33626a802 ("regulator: core: Change voltage setting path")
fails to build  a7348f502ab7 ("regulator: core: Add voltage balancing mechanism")
works           cf6fc8064766 ("regulator: core: Resolve coupled regulators")

Any ideas?

I have also seen regulator issues due to this series:
https://lkml.org/lkml/2018/3/5/731

maintainer
#2 lore raw
Quoted 5 lines · Fabio Estevam
Quoted 2 lines · Tony Lindgren

Looks like with next-20180305 there's a regulator regression where mmc0 won't show any cards or produces errors:

Quoted 8 lines
Quoted 8 lines

mmcblk0: error -110 requesting status
mmc1: new high speed SDIO card at address 0001
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!
print_req_error: I/O error, dev mmcblk0, sector 0 Buffer I/O error on dev mmcblk0, logical block 0, async page read
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!

No other error messages? That seems like there's something going on that's very different to what Fabio was reporting... I'm guessing some voltage application didn't go through but it's hard to tell with so little data. dra7 does seem to have what Fabio had though so there's definitely some effect on the OMAP platforms.

Quoted 2 lines

I have also seen regulator issues due to this series:
https://lkml.org/lkml/2018/3/5/731

Looking at your stuff I'm having trouble figuring out what's going on - we're getting double locking of a parent regulator during enable according to your backtraces but it's not clear to me what took that lock already. regulator_enable() walks the supplies before it takes the lock on the regulator it's immediately being called on, not holding any locks on supplies while enabling. regulator_balance_voltage() then tries to lock the supplies again but lockdep says the lock is already held by regulator_enable(). It's also weird that this doesn't seem to be showing up on other boards in kernelci, the regulator setup on those i.MX boards looks to be quite simple so I'd expect a much wider impact.

I'm wondering if your case is more pain from mutex_lock_nested(), both regulator_lock_coupled() and regulator_lock_supply() will end up using indexes starting at 0 for the locking classes. That doesn't smell right though, but in case my straw clutching works:

If we can't figure it out I'll just drop the series but I'd prefer to at least understand what's going on.

drivers/regulator/core.c +1 -1
index e685f8b94acf..2c5b20a97f51 100644
@@ -159,7 +159,7 @@ static void regulator_lock_supply(struct regulator_dev *rdev)
 {
 	int i;
 
-	for (i = 0; rdev; rdev = rdev_get_supply(rdev), i++)
+	for (i = 1000; rdev; rdev = rdev_get_supply(rdev), i++)
 		mutex_lock_nested(&rdev->mutex, i);
 }
 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180306/8ae451fd/attachment-0001.sig>

#3 lore raw

Hi Mark,

drivers/regulator/core.c static void regulator_lock_supply(struct regulator_dev *rdev) · line 163

Quoted 11 lines · Mark Brown
drivers/regulator/core.c +1 -1
index e685f8b94acf..2c5b20a97f51 100644
@@ -159,7 +159,7 @@ static void regulator_lock_supply(struct regulator_dev *rdev)
 {
        int i;
 
-       for (i = 0; rdev; rdev = rdev_get_supply(rdev), i++)
+       for (i = 1000; rdev; rdev = rdev_get_supply(rdev), i++)
                mutex_lock_nested(&rdev->mutex, i);

With this change the log is a bit different, but still get a kernel hang:
https://pastebin.com/eF08TnuT

Thanks

#4 lore raw
Quoted 12 lines · Fabio Estevam

Hi Mark,

Quoted 8 lines · Mark Brown
drivers/regulator/core.c +0 -0
index e685f8b94acf..2c5b20a97f51 100644
@@ -159,7 +159,7 @@ static void regulator_lock_supply(struct regulator_dev *rdev)
 {
        int i;
 
Quoted 6 lines

drivers/regulator/core.c static void regulator_lock_supply(struct regulator_dev *rdev) · line 163

Quoted 3 lines
drivers/regulator/core.c +1 -1
-       for (i = 0; rdev; rdev = rdev_get_supply(rdev), i++)
+       for (i = 1000; rdev; rdev = rdev_get_supply(rdev), i++)
                mutex_lock_nested(&rdev->mutex, i);

With this change the log is a bit different, but still get a kernel hang:
https://pastebin.com/eF08TnuT

That patch does not seem to change anything for me. The errors I posted yesterday for mmc0 only happen on duovero, other omap variants just fail silently with no cards showing.

Regards,

Tony

maintainer
#5 lore raw
In reply to Fabio Estevam
Quoted 8 lines · Fabio Estevam

drivers/regulator/core.c static void regulator_lock_supply(struct regulator_dev *rdev) · line 163

Quoted 3 lines · Mark Brown
drivers/regulator/core.c +1 -1
-       for (i = 0; rdev; rdev = rdev_get_supply(rdev), i++)
+       for (i = 1000; rdev; rdev = rdev_get_supply(rdev), i++)
                mutex_lock_nested(&rdev->mutex, i);

With this change the log is a bit different, but still get a kernel hang:
https://pastebin.com/eF08TnuT

Yeah, that's what I expected. What we really need to figure out I think
is what exactly is taking the lock that we end up colliding with, I
don't seem to be able to reproduce so I'm having to just go on staring
at the code here.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180306/7015dd76/attachment.sig>

#6 lore raw

Hi Mark,

Quoted 4 lines · Mark Brown

Yeah, that's what I expected. What we really need to figure out I think is what exactly is taking the lock that we end up colliding with, I don't seem to be able to reproduce so I'm having to just go on staring at the code here.

Any debug options I should turn on in order to help us to understand more on the lock issue?

maintainer
#7 lore raw
Quoted 9 lines · Fabio Estevam
Quoted 4 lines · Mark Brown

Yeah, that's what I expected. What we really need to figure out I think is what exactly is taking the lock that we end up colliding with, I don't seem to be able to reproduce so I'm having to just go on staring at the code here.

Any debug options I should turn on in order to help us to understand more on the lock issue?

No, unfortunately, apart from the generic kernel ones (like lockdep).
I'll try to have a poke at adding some tomorrow, though I'm also
considering just dropping the series for now.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180306/591e2d86/attachment.sig>

#8 lore raw
Quoted 3 lines · Mark Brown

No, unfortunately, apart from the generic kernel ones (like lockdep). I'll try to have a poke at adding some tomorrow, though I'm also considering just dropping the series for now.

Ok, lockdep is already enabled by default in the defconfig I am using (imx_v6_v7_defconfig).

Yesterday I spent a few hours trying to understand this locking issue, but didn't get any success on this task.

Thanks

#10 lore raw
In reply to Mark Brown

Hi all,
sorry it took me so long to answer.

Quoted 23 lines · Mark Brown
Quoted 5 lines · Fabio Estevam
Quoted 2 lines · Tony Lindgren

Looks like with next-20180305 there's a regulator regression where mmc0 won't show any cards or produces errors:

Quoted 8 lines
Quoted 8 lines

mmcblk0: error -110 requesting status
mmc1: new high speed SDIO card at address 0001
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!
print_req_error: I/O error, dev mmcblk0, sector 0 Buffer I/O error on dev mmcblk0, logical block 0, async page read
mmcblk0: error -110 requesting status
mmcblk0: recovery failed!

No other error messages? That seems like there's something going on that's very different to what Fabio was reporting... I'm guessing some voltage application didn't go through but it's hard to tell with so little data. dra7 does seem to have what Fabio had though so there's definitely some effect on the OMAP platforms.

Quoted 2 lines
Quoted 1 line

I have also seen regulator issues due to this series:

https://lkml.org/lkml/2018/3/5/731

Quoted 11 lines

Looking at your stuff I'm having trouble figuring out what's going on - we're getting double locking of a parent regulator during enable according to your backtraces but it's not clear to me what took that lock already. regulator_enable() walks the supplies before it takes the lock on the regulator it's immediately being called on, not holding any locks on supplies while enabling. regulator_balance_voltage() then tries to lock the supplies again but lockdep says the lock is already held by regulator_enable(). It's also weird that this doesn't seem to be showing up on other boards in kernelci, the regulator setup on those i.MX boards looks to be quite simple so I'd expect a much wider impact.

I'm trying to figure out what is so special about these boards. The only strange thing, that I haven't noticed at first, is that all regulators share a common supply - dummy regulator. It is defined in anatop_regulator.c.

Quoted 8 lines

I'm wondering if your case is more pain from mutex_lock_nested(), both regulator_lock_coupled() and regulator_lock_supply() will end up using indexes starting at 0 for the locking classes. That doesn't smell right though, but in case my straw clutching works:

If we can't figure it out I'll just drop the series but I'd prefer to at least understand what's going on.

I have been struggling to reproduce the issue on my exynos boards, but all I have achieved is getting the same lockdep warning, but everything else works fine. I think it was a false positive caused by using the same indices in lock_coupled() and lock_supply().

Quoted 6 lines
drivers/regulator/core.c +0 -0
index e685f8b94acf..2c5b20a97f51 100644

drivers/regulator/core.c static void regulator_lock_supply(struct regulator_dev *rdev) · line 165

Quoted 8 lines
drivers/regulator/core.c +1 -1
@@ -159,7 +159,7 @@ static void regulator_lock_supply(struct regulator_dev *rdev)
  {
  	int i;
 
-	for (i = 0; rdev; rdev = rdev_get_supply(rdev), i++)
+	for (i = 1000; rdev; rdev = rdev_get_supply(rdev), i++)
  		mutex_lock_nested(&rdev->mutex, i);
  }
 

Best regards,
Maciej Purski

maintainer
#11 lore raw
Quoted 3 lines · Maciej Purski

I'm trying to figure out what is so special about these boards. The only strange thing, that I haven't noticed at first, is that all regulators share a common supply - dummy regulator. It is defined in anatop_regulator.c.

No, that's a regulator framework thing - the regulator framework will use the dummy regulator as a supply when there's nothing described in the DT so long as the client doesn't explicitly tell it that the supply might be optional.

#12 lore raw
Quoted 11 lines · Mark Brown
Quoted 3 lines · Maciej Purski

I'm trying to figure out what is so special about these boards. The only strange thing, that I haven't noticed at first, is that all regulators share a common supply - dummy regulator. It is defined in anatop_regulator.c.

No, that's a regulator framework thing - the regulator framework will use the dummy regulator as a supply when there's nothing described in the DT so long as the client doesn't explicitly tell it that the supply might be optional.

Ok, thanks for explanation. I think I have found a possibly dangerous scenario, but I can't see this situation possible in Fabio's case.

Assume, that we have a chain of supplies, consisting of at least 3. Say: A->B->C.

When we're setting voltage on A, we lock it, call balance_voltage(), lock suppliers and call set_voltage_rdev(). So we have regulators A, B, C locked. Then set_voltage_rdev() is trying to set voltage of its supply by calling set_voltage_unlocked().

Now we're on the regulator B. Set_voltage_unlocked() calls balance_voltage(), which again locks its supplies, if they exist. B's supply is C, so we end up with having a deadlock on regulator C.

Tony and Fabio, do you find this scenario possible on your boards?

Best regards
Maciej Purski

maintainer
#9 lore raw
In reply to Fabio Estevam
Quoted 2 lines · Fabio Estevam

Yesterday I spent a few hours trying to understand this locking issue, but didn't get any success on this task.

Right, so I'm thinking that if it's taking this much effort to think about the problem we should just drop the patch set for now. I've moved the code to a branch test/coupled in my git so it's not getting in people's way in -next. Maciej, I suspect that you might be able to reproduce this if you set supply names in the regulator definitons for the PMICs on your boards. I think that's the difference here.

#13 lore raw
In reply to Maciej Purski
Quoted 24 lines · Maciej Purski
Quoted 11 lines · Mark Brown
Quoted 3 lines · Maciej Purski

I'm trying to figure out what is so special about these boards. The only strange thing, that I haven't noticed at first, is that all regulators share a common supply - dummy regulator. It is defined in anatop_regulator.c.

No, that's a regulator framework thing - the regulator framework will use the dummy regulator as a supply when there's nothing described in the DT so long as the client doesn't explicitly tell it that the supply might be optional.

Ok, thanks for explanation. I think I have found a possibly dangerous scenario, but I can't see this situation possible in Fabio's case.

Assume, that we have a chain of supplies, consisting of at least 3. Say: A->B->C.

When we're setting voltage on A, we lock it, call balance_voltage(), lock suppliers and call set_voltage_rdev(). So we have regulators A, B, C locked. Then set_voltage_rdev() is trying to set voltage of its supply by calling set_voltage_unlocked().

Quoted 5 lines

Now we're on the regulator B. Set_voltage_unlocked() calls balance_voltage(), which again locks its supplies, if they exist. B's supply is C, so we end up with having a deadlock on regulator C.

Tony and Fabio, do you find this scenario possible on your boards?

For mmc0 at least typically the supply is on the PMIC for omap variants and it's controlled over i2c or spi bus like most other regulators. Then boards some have additional GPIO controlled regulators. So yeah some kind of locking issue between two or more regulators on i2c or spi bus could be the reason.

Regards,

Tony