New l3-noc error with CPUFREQ_DT built-in with v4.0-rc1

DISCUSSION Tony Lindgren 2 participants 10 messages Feb 23, 2015 – Feb 24, 2015

lore

Share & tools

lore

Tony Lindgren

author

2015-02-23 23:59 UTC #0 lore raw

Hi Nishanth,

Olof told me about a new L3 error happening on omap5-uevm with v4.0-rc1:

WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x214/0x340()

4000000.ocp:L3 Custom Error: MASTER MPU TARGET L4PER2 (Idle): Data Access in Supervisor mode during Functional access
...

I tried bisecting this with no luck, but narrowed it down to having CONFIG_CPUFREQ_DT=y causing it, while =m wont' trigger it. This got changed by commit 40d1746d2eee ("ARM:
omap2plus_defconfig: use CONFIG_CPUFREQ_DT").

Any ideas?

Regards,

Tony

Tony Lindgren

author

2015-02-24 01:59 UTC #1 lore raw

Quoted 9 lines · Tony Lindgren

Hi Nishanth,
Olof told me about a new L3 error happening on omap5-uevm with v4.0-rc1:
WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x214/0x340()
4000000.ocp:L3 Custom Error: MASTER MPU TARGET L4PER2 (Idle): Data Access in Supervisor mode during Functional access
...

Quoted 6 lines

I tried bisecting this with no luck, but narrowed it down to having CONFIG_CPUFREQ_DT=y causing it, while =m wont' trigger it. This got changed by commit 40d1746d2eee ("ARM:
omap2plus_defconfig: use CONFIG_CPUFREQ_DT").
Any ideas?

Hmm so setting CONFIG_CPUFREQ_DT=m in arch/arm/configs/omap2plus_defconfig produces the same output with make omap2plus_defconfig as with =y.. So CPUFREQ_DT can't be the real cause of the problem.

It's now looking like the l3-noc warning does not get triggered on every boot.

It also seems the zImage triggering the error does not trigger the error on every boot. To trigger the error, it seems the device needs to be powered down for at least 10 or so seconds between the boots. So far no luck reproducing the error on v3.19.

The easy way to reproduce is to power down omap5 for at least 10 seconds, make omap2lus_defconfig on v4.0-rc1 and boot it.

And so far it looks like next-20150204 works and next-20150209 failed at once so far. But of course I would not trust anything at this point :)

Regards,

Tony

Felipe Balbi

2015-02-24 02:24 UTC #2 lore raw

Hi,

Quoted 29 lines · Tony Lindgren

Quoted 9 lines · Tony Lindgren
Hi Nishanth,
Olof told me about a new L3 error happening on omap5-uevm with v4.0-rc1:
WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x214/0x340()
4000000.ocp:L3 Custom Error: MASTER MPU TARGET L4PER2 (Idle): Data Access in Supervisor mode during Functional access
...
Quoted 6 lines
I tried bisecting this with no luck, but narrowed it down to having CONFIG_CPUFREQ_DT=y causing it, while =m wont' trigger it. This got changed by commit 40d1746d2eee ("ARM:
omap2plus_defconfig: use CONFIG_CPUFREQ_DT").
Any ideas?
Hmm so setting CONFIG_CPUFREQ_DT=m in arch/arm/configs/omap2plus_defconfig produces the same output with make omap2plus_defconfig as with =y.. So CPUFREQ_DT can't be the real cause of the problem.
It's now looking like the l3-noc warning does not get triggered on every boot.
It also seems the zImage triggering the error does not trigger the error on every boot. To trigger the error, it seems the device needs to be powered down for at least 10 or so seconds between the boots. So far no luck reproducing the error on v3.19.

Quoted 6 lines

The easy way to reproduce is to power down omap5 for at least 10 seconds, make omap2lus_defconfig on v4.0-rc1 and boot it.
And so far it looks like next-20150204 works and next-20150209 failed at once so far. But of course I would not trust anything at this point :)

got a log of the failure ? Is it pointing to a device or one of the L4s?

Might be worth to boot with just the bare minimum (UART & timers) and disable everything else. You might need to build busybox and append that to the kernel so you don't need to rely on MMC/USB/etc for rootfs.

After that, you could start enabling modules one by one (as modules, not built-in) and loading them one by one to see which one causes the failure. Big PITA, I know, but I can't think of any other way to go about this.

Signature · 8 lines

balbi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20150223/26c84f17/attachment.sig>

Tony Lindgren

author

2015-02-24 02:35 UTC #3 lore raw

Quoted 32 lines · Felipe Balbi

Hi,
Quoted 29 lines · Tony Lindgren
Quoted 9 lines · Tony Lindgren
Hi Nishanth,
Olof told me about a new L3 error happening on omap5-uevm with v4.0-rc1:
WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x214/0x340()
4000000.ocp:L3 Custom Error: MASTER MPU TARGET L4PER2 (Idle): Data Access in Supervisor mode during Functional access
...
Quoted 6 lines
I tried bisecting this with no luck, but narrowed it down to having CONFIG_CPUFREQ_DT=y causing it, while =m wont' trigger it. This got changed by commit 40d1746d2eee ("ARM:
omap2plus_defconfig: use CONFIG_CPUFREQ_DT").
Any ideas?
Hmm so setting CONFIG_CPUFREQ_DT=m in arch/arm/configs/omap2plus_defconfig produces the same output with make omap2plus_defconfig as with =y.. So CPUFREQ_DT can't be the real cause of the problem.
It's now looking like the l3-noc warning does not get triggered on every boot.
It also seems the zImage triggering the error does not trigger the error on every boot. To trigger the error, it seems the device needs to be powered down for at least 10 or so seconds between the boots. So far no luck reproducing the error on v3.19.

Quoted 8 lines

Quoted 6 lines
The easy way to reproduce is to power down omap5 for at least 10 seconds, make omap2lus_defconfig on v4.0-rc1 and boot it.
And so far it looks like next-20150204 works and next-20150209 failed at once so far. But of course I would not trust anything at this point :)
got a log of the failure ? Is it pointing to a device or one of the L4s?

Well mostly the MASTER MPU TARGET L4PER2, the following stack dump is really the stack dump of the l3_interrupt_handler.

Quoted 8 lines

Might be worth to boot with just the bare minimum (UART & timers) and disable everything else. You might need to build busybox and append that to the kernel so you don't need to rely on MMC/USB/etc for rootfs.
After that, you could start enabling modules one by one (as modules, not built-in) and loading them one by one to see which one causes the failure. Big PITA, I know, but I can't think of any other way to go about this.

It seems the best way to deal with this is to make the l3_handle_target actually show the address where the error happened to limit it down to a single device..

Regards,

Tony

Tony Lindgren

author

2015-02-24 03:01 UTC #4 lore raw

Quoted 49 lines · Tony Lindgren

Quoted 32 lines · Felipe Balbi
Hi,
Quoted 29 lines · Tony Lindgren
Quoted 9 lines · Tony Lindgren
Hi Nishanth,
Olof told me about a new L3 error happening on omap5-uevm with v4.0-rc1:
WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x214/0x340()
4000000.ocp:L3 Custom Error: MASTER MPU TARGET L4PER2 (Idle): Data Access in Supervisor mode during Functional access
...
Quoted 6 lines
I tried bisecting this with no luck, but narrowed it down to having CONFIG_CPUFREQ_DT=y causing it, while =m wont' trigger it. This got changed by commit 40d1746d2eee ("ARM:
omap2plus_defconfig: use CONFIG_CPUFREQ_DT").
Any ideas?
Hmm so setting CONFIG_CPUFREQ_DT=m in arch/arm/configs/omap2plus_defconfig produces the same output with make omap2plus_defconfig as with =y.. So CPUFREQ_DT can't be the real cause of the problem.
It's now looking like the l3-noc warning does not get triggered on every boot.
It also seems the zImage triggering the error does not trigger the error on every boot. To trigger the error, it seems the device needs to be powered down for at least 10 or so seconds between the boots. So far no luck reproducing the error on v3.19.
Quoted 8 lines
Quoted 6 lines
The easy way to reproduce is to power down omap5 for at least 10 seconds, make omap2lus_defconfig on v4.0-rc1 and boot it.
And so far it looks like next-20150204 works and next-20150209 failed at once so far. But of course I would not trust anything at this point :)
got a log of the failure ? Is it pointing to a device or one of the L4s?
Well mostly the MASTER MPU TARGET L4PER2, the following stack dump is really the stack dump of the l3_interrupt_handler.
Quoted 4 lines
Might be worth to boot with just the bare minimum (UART & timers) and disable everything else. You might need to build busybox and append that to the kernel so you don't need to rely on MMC/USB/etc for rootfs.

Quoted 8 lines

Quoted 4 lines
After that, you could start enabling modules one by one (as modules, not built-in) and loading them one by one to see which one causes the failure. Big PITA, I know, but I can't think of any other way to go about this.
It seems the best way to deal with this is to make the l3_handle_target actually show the address where the error happened to limit it down to a single device..

Looks like the address is 0 for "Custom Error". Anyways, reverting a fix for similar issue found on omap3 so far seems to help, that's 3d009c8c61f9 ("gpio: omap: Fix bad device access with setup_irq()").

That got merged very late for v3.19, so it certainly explains why it was not noticed until now.

Regards,

Tony

Felipe Balbi

2015-02-24 03:12 UTC #9 lore raw

In reply to Tony Lindgren

Quoted 49 lines · Tony Lindgren

Quoted 32 lines · Felipe Balbi
Hi,
Quoted 29 lines · Tony Lindgren
Quoted 9 lines · Tony Lindgren
Hi Nishanth,
Olof told me about a new L3 error happening on omap5-uevm with v4.0-rc1:
WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x214/0x340()
4000000.ocp:L3 Custom Error: MASTER MPU TARGET L4PER2 (Idle): Data Access in Supervisor mode during Functional access
...
Quoted 6 lines
I tried bisecting this with no luck, but narrowed it down to having CONFIG_CPUFREQ_DT=y causing it, while =m wont' trigger it. This got changed by commit 40d1746d2eee ("ARM:
omap2plus_defconfig: use CONFIG_CPUFREQ_DT").
Any ideas?
Hmm so setting CONFIG_CPUFREQ_DT=m in arch/arm/configs/omap2plus_defconfig produces the same output with make omap2plus_defconfig as with =y.. So CPUFREQ_DT can't be the real cause of the problem.
It's now looking like the l3-noc warning does not get triggered on every boot.
It also seems the zImage triggering the error does not trigger the error on every boot. To trigger the error, it seems the device needs to be powered down for at least 10 or so seconds between the boots. So far no luck reproducing the error on v3.19.
Quoted 8 lines
Quoted 6 lines
The easy way to reproduce is to power down omap5 for at least 10 seconds, make omap2lus_defconfig on v4.0-rc1 and boot it.
And so far it looks like next-20150204 works and next-20150209 failed at once so far. But of course I would not trust anything at this point :)
got a log of the failure ? Is it pointing to a device or one of the L4s?
Well mostly the MASTER MPU TARGET L4PER2, the following stack dump is really the stack dump of the l3_interrupt_handler.
Quoted 4 lines
Might be worth to boot with just the bare minimum (UART & timers) and disable everything else. You might need to build busybox and append that to the kernel so you don't need to rely on MMC/USB/etc for rootfs.

Quoted 8 lines

Quoted 4 lines
After that, you could start enabling modules one by one (as modules, not built-in) and loading them one by one to see which one causes the failure. Big PITA, I know, but I can't think of any other way to go about this.
It seems the best way to deal with this is to make the l3_handle_target actually show the address where the error happened to limit it down to a single device..

you can't really do that from within l3. It doesn't have enough information to figure that out. Since it pointed you to l4per2, then you need to decode l4per2's debug registers. That has never been implemented, though. What happened here is that l4per2 detected the bogus access from one of the devices attached to it and passed the error up to l3. Since we only have l3 decoding, that's what you see and it ends up being really cryptic.

If you decode l4per2's registers, I'm sure it'll point you to a real device. I guess just to prove the concept, you just hack it inside l3 irq handler, though ideally we would have a real drivers/bus/omap-l4.c, or something like that.

Signature · 8 lines

balbi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20150223/39e82c45/attachment-0001.sig>

Felipe Balbi

2015-02-24 03:15 UTC #5 lore raw

In reply to Tony Lindgren

Hi,

Quoted 55 lines · Tony Lindgren

Quoted 54 lines · Tony Lindgren
Quoted 32 lines · Felipe Balbi
Hi,
Quoted 29 lines · Tony Lindgren
Quoted 9 lines · Tony Lindgren
Hi Nishanth,
Olof told me about a new L3 error happening on omap5-uevm with v4.0-rc1:
WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x214/0x340()
4000000.ocp:L3 Custom Error: MASTER MPU TARGET L4PER2 (Idle): Data Access in Supervisor mode during Functional access
...
Quoted 6 lines
I tried bisecting this with no luck, but narrowed it down to having CONFIG_CPUFREQ_DT=y causing it, while =m wont' trigger it. This got changed by commit 40d1746d2eee ("ARM:
omap2plus_defconfig: use CONFIG_CPUFREQ_DT").
Any ideas?
Hmm so setting CONFIG_CPUFREQ_DT=m in arch/arm/configs/omap2plus_defconfig produces the same output with make omap2plus_defconfig as with =y.. So CPUFREQ_DT can't be the real cause of the problem.
It's now looking like the l3-noc warning does not get triggered on every boot.
It also seems the zImage triggering the error does not trigger the error on every boot. To trigger the error, it seems the device needs to be powered down for at least 10 or so seconds between the boots. So far no luck reproducing the error on v3.19.
Quoted 8 lines
Quoted 6 lines
The easy way to reproduce is to power down omap5 for at least 10 seconds, make omap2lus_defconfig on v4.0-rc1 and boot it.
And so far it looks like next-20150204 works and next-20150209 failed at once so far. But of course I would not trust anything at this point :)
got a log of the failure ? Is it pointing to a device or one of the L4s?
Well mostly the MASTER MPU TARGET L4PER2, the following stack dump is really the stack dump of the l3_interrupt_handler.
Quoted 8 lines
Might be worth to boot with just the bare minimum (UART & timers) and disable everything else. You might need to build busybox and append that to the kernel so you don't need to rely on MMC/USB/etc for rootfs.
After that, you could start enabling modules one by one (as modules, not built-in) and loading them one by one to see which one causes the failure. Big PITA, I know, but I can't think of any other way to go about this.

Quoted 5 lines

Quoted 3 lines
It seems the best way to deal with this is to make the l3_handle_target actually show the address where the error happened to limit it down to a single device..
Looks like the address is 0 for "Custom Error". Anyways, reverting

yeah, that's because the error comes from l4per2, not l3 :-)

Quoted 2 lines

a fix for similar issue found on omap3 so far seems to help, that's 3d009c8c61f9 ("gpio: omap: Fix bad device access with setup_irq()").

if we revert that, we regress omap3, right ?

Signature · 8 lines

balbi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20150223/ddbd7a61/attachment.sig>

Tony Lindgren

author

2015-02-24 03:21 UTC #6 lore raw

Quoted 6 lines · Felipe Balbi

Quoted 3 lines · Tony Lindgren
* Tony Lindgren <[email protected]> [150223 18:43]:
Looks like the address is 0 for "Custom Error". Anyways, reverting
yeah, that's because the error comes from l4per2, not l3 :-)

Right so it seems :)

Quoted 4 lines

Quoted 2 lines
a fix for similar issue found on omap3 so far seems to help, that's 3d009c8c61f9 ("gpio: omap: Fix bad device access with setup_irq()").
if we revert that, we regress omap3, right ?

Yes we saw that getting triggered on omap3/4 before 3d009c8c61f9.

Now looking at the stack trace again, it actually has something:

...
[<c05999a4>] (__irq_svc) from [<c0599164>] (_raw_spin_unlock_irqrestore+0x34/0x44) [<c0599164>] (_raw_spin_unlock_irqrestore) from [<c0027120>] (omap_hwmod_idle+0x34/0x44) [<c0027120>] (omap_hwmod_idle) from [<c00283f8>] (omap_device_idle+0x38/0x78) [<c00283f8>] (omap_device_idle) from [<c0028454>] (_od_runtime_suspend+0x1c/0x24) [<c0028454>] (_od_runtime_suspend) from [<c03b5214>] (__rpm_callback+0x2c/0x60) [<c03b5214>] (__rpm_callback) from [<c03b5268>] (rpm_callback+0x20/0x80) [<c03b5268>] (rpm_callback) from [<c03b56b8>] (rpm_suspend+0xe8/0x55c) [<c03b56b8>] (rpm_suspend) from [<c03b6bdc>] (pm_runtime_work+0x74/0xa8) [<c03b6bdc>] (pm_runtime_work) from [<c0054608>] (process_one_work+0x1b0/0x4a0)
...

If it's the gpio-omap, there's probably some confusion still in the driver regarding the GPIO bank idle status. Anyways, will look more into it tomorrow.

Regards,

Tony

Tony Lindgren

author

2015-02-24 03:24 UTC #7 lore raw

Quoted 31 lines · Tony Lindgren

Quoted 6 lines · Felipe Balbi
Quoted 3 lines · Tony Lindgren
* Tony Lindgren <[email protected]> [150223 18:43]:
Looks like the address is 0 for "Custom Error". Anyways, reverting
yeah, that's because the error comes from l4per2, not l3 :-)
Right so it seems :)
Quoted 4 lines
Quoted 2 lines
a fix for similar issue found on omap3 so far seems to help, that's 3d009c8c61f9 ("gpio: omap: Fix bad device access with setup_irq()").
if we revert that, we regress omap3, right ?
Yes we saw that getting triggered on omap3/4 before 3d009c8c61f9.
Now looking at the stack trace again, it actually has something:
...
[<c05999a4>] (__irq_svc) from [<c0599164>] (_raw_spin_unlock_irqrestore+0x34/0x44) [<c0599164>] (_raw_spin_unlock_irqrestore) from [<c0027120>] (omap_hwmod_idle+0x34/0x44) [<c0027120>] (omap_hwmod_idle) from [<c00283f8>] (omap_device_idle+0x38/0x78) [<c00283f8>] (omap_device_idle) from [<c0028454>] (_od_runtime_suspend+0x1c/0x24) [<c0028454>] (_od_runtime_suspend) from [<c03b5214>] (__rpm_callback+0x2c/0x60) [<c03b5214>] (__rpm_callback) from [<c03b5268>] (rpm_callback+0x20/0x80) [<c03b5268>] (rpm_callback) from [<c03b56b8>] (rpm_suspend+0xe8/0x55c) [<c03b56b8>] (rpm_suspend) from [<c03b6bdc>] (pm_runtime_work+0x74/0xa8) [<c03b6bdc>] (pm_runtime_work) from [<c0054608>] (process_one_work+0x1b0/0x4a0)
...

Quoted 3 lines

If it's the gpio-omap, there's probably some confusion still in the driver regarding the GPIO bank idle status. Anyways, will look more into it tomorrow.

And now I'm seeing the error with 3d009c8c61f9 reverted, so it does not seem to be that either..

Regards,

Tony

Felipe Balbi

2015-02-24 14:46 UTC #8 lore raw

Hi,

Quoted 32 lines · Tony Lindgren

Quoted 31 lines · Tony Lindgren
Quoted 6 lines · Felipe Balbi
Quoted 3 lines · Tony Lindgren
* Tony Lindgren <[email protected]> [150223 18:43]:
Looks like the address is 0 for "Custom Error". Anyways, reverting
yeah, that's because the error comes from l4per2, not l3 :-)
Right so it seems :)
Quoted 4 lines
Quoted 2 lines
a fix for similar issue found on omap3 so far seems to help, that's 3d009c8c61f9 ("gpio: omap: Fix bad device access with setup_irq()").
if we revert that, we regress omap3, right ?
Yes we saw that getting triggered on omap3/4 before 3d009c8c61f9.
Now looking at the stack trace again, it actually has something:
...
[<c05999a4>] (__irq_svc) from [<c0599164>] (_raw_spin_unlock_irqrestore+0x34/0x44) [<c0599164>] (_raw_spin_unlock_irqrestore) from [<c0027120>] (omap_hwmod_idle+0x34/0x44) [<c0027120>] (omap_hwmod_idle) from [<c00283f8>] (omap_device_idle+0x38/0x78) [<c00283f8>] (omap_device_idle) from [<c0028454>] (_od_runtime_suspend+0x1c/0x24) [<c0028454>] (_od_runtime_suspend) from [<c03b5214>] (__rpm_callback+0x2c/0x60) [<c03b5214>] (__rpm_callback) from [<c03b5268>] (rpm_callback+0x20/0x80) [<c03b5268>] (rpm_callback) from [<c03b56b8>] (rpm_suspend+0xe8/0x55c) [<c03b56b8>] (rpm_suspend) from [<c03b6bdc>] (pm_runtime_work+0x74/0xa8) [<c03b6bdc>] (pm_runtime_work) from [<c0054608>] (process_one_work+0x1b0/0x4a0)
...

Quoted 6 lines

Quoted 3 lines
If it's the gpio-omap, there's probably some confusion still in the driver regarding the GPIO bank idle status. Anyways, will look more into it tomorrow.
And now I'm seeing the error with 3d009c8c61f9 reverted, so it does not seem to be that either..

well, that's good :-)

Signature · 8 lines

balbi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20150224/9fbfc700/attachment.sig>