Panda ES board hang when using GPIO as interrupt

Hi Kevin, Tarun,

We are using the expansion connector A on Panda board to mount a SDIO WiFi dongle on MMC2 with a level triggered interrupt signal connected to GPIO
138. It's been working fine until 3.5 rc1. The board hang randomly within 5 mins during a network traffic test. After bisecting we found the culprit is "[PATCH 8/8] gpio/omap: fix missing check in *_runtime_suspend()" [1].

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Any thoughts and suggestions?

From 0e1b322451b7a49487d2d17a147db1aa1d1119fa Mon Sep 17 00:00:00 2001

Thanks,
Franky

[1] http://article.gmane.org/gmane.linux.ports.arm.omap/75708/

Quoted 15 lines · Franky Lin

Hi Kevin, Tarun,

We are using the expansion connector A on Panda board to mount a SDIO WiFi dongle on MMC2 with a level triggered interrupt signal connected to GPIO
138. It's been working fine until 3.5 rc1. The board hang randomly within 5 mins during a network traffic test. After bisecting we found the culprit is "[PATCH 8/8] gpio/omap: fix missing check in *_runtime_suspend()" [1].

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Quoted 1 line

Any thoughts and suggestions?

I just had a quick look at the code. Can you please check if the attached patch solves
the issue? I just boot tested on Panda and Blaze.
--
Tarun

Quoted 23 lines · Tarun Kanti

Quoted 21 lines · Franky Lin

Hi Kevin, Tarun,

We are using the expansion connector A on Panda board to mount a SDIO WiFi
dongle on MMC2 with a level triggered interrupt signal connected to GPIO
138. It's been working fine until 3.5 rc1. The board hang randomly within 5
mins during a network traffic test. After bisecting we found the culprit is
"[PATCH 8/8] gpio/omap: fix missing check in *_runtime_suspend()" [1].

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two
patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline
kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Quoted 8 lines

Quoted 1 line

Any thoughts and suggestions?

I just had a quick look at the code. Can you please check if the attached patch solves
the issue? I just boot tested on Panda and Blaze.
--
Tarun

Thanks for the prompt reply.

Booting is fine even without the patch and revert. The wifi dongle generates interrupt whenever there is data packet available for host to read. So during a traffic test a significant numbers of interrupt will be triggered through the GPIO. So I assume it has something to do with the interrupt GPIO.

With the patch, the kernel still crashes. But the symptom is slightly different. Now it has a panic log every time. See attachment.

Regards,
Franky

Hi Kevin, Tarun,

We are using the expansion connector A on Panda board to mount a SDIO WiFi dongle on MMC2 with a level triggered interrupt signal connected to GPIO 138. It's been working fine until 3.5 rc1. The board hang randomly within 5 mins during a network traffic test. After bisecting we found the culprit is "[PATCH 8/8] gpio/omap: fix missing check in *_runtime_suspend()" [1].

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Any thoughts and suggestions?

Quoted 9 lines · Franky Lin

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Any thoughts and suggestions?

If reverting the patch fixes your problem, can you isolate down to which part of that patch causes the problem? IOW, can you fix your problem if you undo just the hunk added in runtime_suspend or undo just the moved hunk runtime_resume? Or is reverting both required?

I suspect the added runtime_suspend hunk is causing the problems, so can you see if just undoing that part works[1]. If that works, I will give a bit more of a thinking on it tomorrow.

Thanks for reporting the problem! Bug reports like this that have
clearly been thoroughly researched and bisected are greatly appreciated!

Kevin

Quoted 16 lines · Kevin Hilman

Quoted 9 lines · Franky Lin

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Any thoughts and suggestions?

If reverting the patch fixes your problem, can you isolate down to which part of that patch causes the problem? IOW, can you fix your problem if you undo just the hunk added in runtime_suspend or undo just the moved hunk runtime_resume? Or is reverting both required?

Quoted 3 lines

I suspect the added runtime_suspend hunk is causing the problems, so can you see if just undoing that part works[1]. If that works, I will give a bit more of a thinking on it tomorrow.

runtime_suspend hunk is fine. The hang still exist after reverting it. The culprit is the moved hunk in runtime_resume. Reverting it makes the hang disappear.

Quoted 17 lines · Franky Lin

Quoted 16 lines · Kevin Hilman

Quoted 9 lines · Franky Lin

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Any thoughts and suggestions?

If reverting the patch fixes your problem, can you isolate down to which part of that patch causes the problem? IOW, can you fix your problem if you undo just the hunk added in runtime_suspend or undo just the moved hunk runtime_resume? Or is reverting both required?

Quoted 7 lines

Quoted 3 lines

I suspect the added runtime_suspend hunk is causing the problems, so can you see if just undoing that part works[1]. If that works, I will give a bit more of a thinking on it tomorrow.

runtime_suspend hunk is fine. The hang still exist after reverting it. The culprit is the moved hunk in runtime_resume. Reverting it makes the hang disappear.

Thanks. From reviewing the code the only thing that appears suspect based upon your findings is the return if we find the context has not been lost. We are not checking if "workaround_enabled" is set before we return.

Could you try the following change on top of v3.5-rc3?

Also, could you add a print in the runtime_suspend/resume() functions so we can see how often these are being called. In my case, I really don't see these being exercised and I am wondering how often you see suspend/resume being called in your setup.

Quoted 35 lines · Jon Hunter

Quoted 19 lines · Franky Lin

Quoted 18 lines · Kevin Hilman

Quoted 9 lines · Franky Lin

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Any thoughts and suggestions?

If reverting the patch fixes your problem, can you isolate down to which
part of that patch causes the problem? IOW, can you fix your problem if
you undo just the hunk added in runtime_suspend or undo just the moved hunk runtime_resume? Or is reverting both required?

Quoted 8 lines

Quoted 4 lines

I suspect the added runtime_suspend hunk is causing the problems, so can
you see if just undoing that part works[1]. If that works, I will give a bit more of a thinking on it tomorrow.

runtime_suspend hunk is fine. The hang still exist after reverting it. The culprit is the moved hunk in runtime_resume. Reverting it makes the hang disappear.

Thanks. From reviewing the code the only thing that appears suspect based upon your findings is the return if we find the context has not been lost.
We are not checking if "workaround_enabled" is set before we return.

Quoted 2 lines

Could you try the following change on top of v3.5-rc3?

The patch doesn't help. And I also managed to probe the signal. It's active when it hung.

Quoted 5 lines

Also, could you add a print in the runtime_suspend/resume() functions so we can see how often these are being called. In my case, I really don't see
these being exercised and I am wondering how often you see suspend/resume being called in your setup.

Well, the runtime_suspend/resume never get called during the test.

Ok. Any way to manually reset the wlan module to deactivate the gpio when it is hung? I am wondering if the gpio is deactivated if the board comes back to life, indicating it is stuck in the interrupt somewhere.

Well, at least that is consistent with what I see, but also perplexing that it takes sometime to fail. Can you try the following as a debug patch to see if it is in the context restore that is the problem. From your testing and bisect, the only possible difference in the current kernel is that it could perform the context restore when acquiring the gpio.

drivers/gpio/gpio-omap.c +0 -0

index c4ed172..a2401bd 100644

Quoted 3 lines · Jon Hunter

Ok. Any way to manually reset the wlan module to deactivate the gpio when it is hung? I am wondering if the gpio is deactivated if the board comes back to life, indicating it is stuck in the interrupt somewhere.

The only way I can think of is removing the module manually. But it didn't bring the board back to live.

Quoted 7 lines

Well, at least that is consistent with what I see, but also perplexing that it takes sometime to fail. Can you try the following as a debug patch to see if it is in the context restore that is the problem. From your testing and bisect, the only possible difference in the current kernel is that it could perform the context restore when acquiring the gpio.

Quoted 6 lines

drivers/gpio/gpio-omap.c +0 -0

index c4ed172..a2401bd 100644

drivers/gpio/gpio-omap.c void omap2_gpio_resume_after_idle(void) · line 1348

Quoted 8 lines

drivers/gpio/gpio-omap.c +2 -0

@@ -1341,6 +1341,8 @@ void omap2_gpio_resume_after_idle(void)
  #if defined(CONFIG_PM_RUNTIME)
  static void omap_gpio_restore_context(struct gpio_bank *bank)
  {
+       return;
+
         __raw_writel(bank->context.wake_en,
                                 bank->base + bank->regs->wkup_en);
         __raw_writel(bank->context.ctrl, bank->base + bank->regs->ctrl);

This one works! It can run more than 20 mins.

I found one interesting thing. When I added the print info to see when runtime_suspend/resume get called, it seems like the suspend/resume is unbalance during boot. Resume got called more than suspend. So I hack the code to make sure suspend and resume are called in pair. A resume without suspend will do nothing and return immediately. This also makes the hang vanish.

Quoted 6 lines · Franky Lin

I found one interesting thing. When I added the print info to see when runtime_suspend/resume get called, it seems like the suspend/resume is unbalance during boot. Resume got called more than suspend. So I hack the code to make sure suspend and resume are called in pair. A resume without suspend will do nothing and return immediately. This also makes the hang vanish.

I am not 100% sure I follow. On boot I would expect to see a resume/suspend due to the probe on the irq bank and then I would expect to see another resume from the acquisition of the gpio, however, I would not expect a suspend until the gpio is freed, which I don't believe you are doing.

Can you share your hack? Just paste the diff? This may help me understand more.

Quoted 9 lines · Jon Hunter

Quoted 6 lines · Franky Lin

I found one interesting thing. When I added the print info to see when runtime_suspend/resume get called, it seems like the suspend/resume is unbalance during boot. Resume got called more than suspend. So I hack the code to make sure suspend and resume are called in pair. A resume without suspend will do nothing and return immediately. This also makes the hang vanish.

Quoted 9 lines

I am not 100% sure I follow. On boot I would expect to see a resume/suspend due to the probe on the irq bank and then I would expect to see another resume from the acquisition of the gpio, however, I would not expect a suspend until the gpio is freed, which I don't believe you are doing.

Can you share your hack? Just paste the diff? This may help me understand more.

OK.
This is what I saw in the log:

[    0.171844] dummy:
[    0.172912] NET: Registered protocol family 16
[    0.173431] GPMC revision 6.0
[    0.173492] gpmc: irq-52 could not claim: err -22
[    0.177551] ??????omap_gpio_runtime_resume
[    0.178619] OMAP GPIO hardware version 0.1
[    0.178649] !!!!!omap_gpio_runtime_suspend
[    0.178771] ??????omap_gpio_runtime_resume
[    0.179351] !!!!!omap_gpio_runtime_suspend

[    0.179504] ??????omap_gpio_runtime_resume
[    0.180023] !!!!!omap_gpio_runtime_suspend
[    0.180145] ??????omap_gpio_runtime_resume
[    0.180694] !!!!!omap_gpio_runtime_suspend
[    0.180847] ??????omap_gpio_runtime_resume
[    0.181365] !!!!!omap_gpio_runtime_suspend
[    0.181518] ??????omap_gpio_runtime_resume
[    0.182037] !!!!!omap_gpio_runtime_suspend

[    0.185089] omap_mux_init: Add partition: #1: core, flags: 2
[    0.186462] omap_mux_init: Add partition: #2: wkup, flags: 2
[    0.186584] error setting wl12xx data: -38
[    0.189788] _omap_mux_get_by_name: Could not find signal
uart1_rx.uart1_rx
[    0.189788] _omap_mux_get_by_name: Could not find signal
uart1_rx.uart1_rx
[    0.239501] ??????omap_gpio_runtime_resume
[    0.239532] ??????omap_gpio_runtime_resume
[    0.241058]  usbhs_omap: alias fck already exists
[    0.244781] ??????omap_gpio_runtime_resume

drivers/gpio/gpio-omap.c +1 -1

index c4ed172..bca3985 100644
@@ -1146,7 +1146,7 @@ static int __devinit omap_gpio_probe(struct
 platform_device *pdev)
 
 #if defined(CONFIG_PM_RUNTIME)
 static void omap_gpio_restore_context(struct gpio_bank *bank);
-
+static int flag = 0;
 static int omap_gpio_runtime_suspend(struct device *dev)
 {

        struct platform_device *pdev = to_platform_device(dev);

drivers/gpio/gpio-omap.c +2 -0

@@ -1155,6 +1155,8 @@ static int omap_gpio_runtime_suspend(struct device
 *dev)
        unsigned long flags;
        u32 wake_low, wake_hi;
 
+       flag ++;
+
        spin_lock_irqsave(&bank->lock, flags);
 

        /*

drivers/gpio/gpio-omap.c +0 -0

@@ -1221,6 +1223,11 @@ static int omap_gpio_runtime_resume(struct device
 *dev)

Quoted 9 lines · Jon Hunter

Quoted 6 lines · Franky Lin

I found one interesting thing. When I added the print info to see when runtime_suspend/resume get called, it seems like the suspend/resume is unbalance during boot. Resume got called more than suspend. So I hack the code to make sure suspend and resume are called in pair. A resume without suspend will do nothing and return immediately. This also makes the hang vanish.

Quoted 9 lines

I am not 100% sure I follow. On boot I would expect to see a resume/suspend due to the probe on the irq bank and then I would expect to see another resume from the acquisition of the gpio, however, I would not expect a suspend until the gpio is freed, which I don't believe you are doing.

Can you share your hack? Just paste the diff? This may help me understand more.

OK.
This is what I saw in the log:

[    0.171844] dummy:
[    0.172912] NET: Registered protocol family 16
[    0.173431] GPMC revision 6.0
[    0.173492] gpmc: irq-52 could not claim: err -22
[    0.177551] ??????omap_gpio_runtime_resume
[    0.178619] OMAP GPIO hardware version 0.1
[    0.178649] !!!!!omap_gpio_runtime_suspend
[    0.178771] ??????omap_gpio_runtime_resume
[    0.179351] !!!!!omap_gpio_runtime_suspend
[    0.179504] ??????omap_gpio_runtime_resume
[    0.180023] !!!!!omap_gpio_runtime_suspend
[    0.180145] ??????omap_gpio_runtime_resume
[    0.180694] !!!!!omap_gpio_runtime_suspend
[    0.180847] ??????omap_gpio_runtime_resume
[    0.181365] !!!!!omap_gpio_runtime_suspend
[    0.181518] ??????omap_gpio_runtime_resume
[    0.182037] !!!!!omap_gpio_runtime_suspend
[    0.185089] omap_mux_init: Add partition: #1: core, flags: 2
[    0.186462] omap_mux_init: Add partition: #2: wkup, flags: 2
[    0.186584] error setting wl12xx data: -38

[    0.189788] _omap_mux_get_by_name: Could not find signal
uart1_rx.uart1_rx
[    0.189788] _omap_mux_get_by_name: Could not find signal
uart1_rx.uart1_rx
[    0.239501] ??????omap_gpio_runtime_resume
[    0.239532] ??????omap_gpio_runtime_resume
[    0.241058]  usbhs_omap: alias fck already exists
[    0.244781] ??????omap_gpio_runtime_resume

Quoted 9 lines · Jon Hunter

Quoted 6 lines · Franky Lin

I found one interesting thing. When I added the print info to see when runtime_suspend/resume get called, it seems like the suspend/resume is unbalance during boot. Resume got called more than suspend. So I hack the code to make sure suspend and resume are called in pair. A resume without suspend will do nothing and return immediately. This also makes the hang vanish.

Quoted 9 lines

I am not 100% sure I follow. On boot I would expect to see a resume/suspend due to the probe on the irq bank and then I would expect to see another resume from the acquisition of the gpio, however, I would not expect a suspend until the gpio is freed, which I don't believe you are doing.

Can you share your hack? Just paste the diff? This may help me understand more.

OK.
This is what I saw in the log:

[    0.171844] dummy:
[    0.172912] NET: Registered protocol family 16
[    0.173431] GPMC revision 6.0
[    0.173492] gpmc: irq-52 could not claim: err -22
[    0.177551] ??????omap_gpio_runtime_resume
[    0.178619] OMAP GPIO hardware version 0.1
[    0.178649] !!!!!omap_gpio_runtime_suspend
[    0.178771] ??????omap_gpio_runtime_resume
[    0.179351] !!!!!omap_gpio_runtime_suspend
[    0.179504] ??????omap_gpio_runtime_resume
[    0.180023] !!!!!omap_gpio_runtime_suspend
[    0.180145] ??????omap_gpio_runtime_resume
[    0.180694] !!!!!omap_gpio_runtime_suspend
[    0.180847] ??????omap_gpio_runtime_resume
[    0.181365] !!!!!omap_gpio_runtime_suspend
[    0.181518] ??????omap_gpio_runtime_resume
[    0.182037] !!!!!omap_gpio_runtime_suspend
[    0.185089] omap_mux_init: Add partition: #1: core, flags: 2
[    0.186462] omap_mux_init: Add partition: #2: wkup, flags: 2
[    0.186584] error setting wl12xx data: -38

[    0.189788] _omap_mux_get_by_name: Could not find signal
uart1_rx.uart1_rx
[    0.189788] _omap_mux_get_by_name: Could not find signal
uart1_rx.uart1_rx
[    0.239501] ??????omap_gpio_runtime_resume
[    0.239532] ??????omap_gpio_runtime_resume
[    0.241058]  usbhs_omap: alias fck already exists
[    0.244781] ??????omap_gpio_runtime_resume

I am wondering if this could be the bug ... on start-up I see that we do a context restore on bank1 during the probe which is before we have done the first suspend! In other words, we could restore a bad/uninitialised context for bank1. In the case of bank1, the loss count starts at 1 and not 0 and so we falsely think we need to perform a restore :-(

[    0.176269] omap_gpio_runtime_resume: bank @ 0xfc310000
[    0.177276] omap_gpio_runtime_resume: count 0, now 1
[    0.177276] gpiochip_add: registered GPIOs 0 to 31 on device: gpio
[    0.177642] omap_gpio_runtime_suspend: bank @ 0xfc310000

Can you try ...

drivers/gpio/gpio-omap.c +3 -0

index c4ed172..9623408 100644
@@ -1086,6 +1086,9 @@ static int __devinit omap_gpio_probe(struct
 platform_device *pdev)
  #ifdef CONFIG_OF_GPIO
         bank->chip.of_node = of_node_get(node);
  #endif
+       if (bank->get_context_loss_count)
+               bank->context_loss_count =
+                               bank->get_context_loss_count(bank->dev);
 
         bank->irq_base = irq_alloc_descs(-1, 0, bank->width, 0);

         if (bank->irq_base < 0) {

Quoted 27 lines · Jon Hunter

I am wondering if this could be the bug ... on start-up I see that we do a context restore on bank1 during the probe which is before we have done the first suspend! In other words, we could restore a bad/uninitialised context for bank1. In the case of bank1, the loss count starts at 1 and not 0 and so we falsely think we need to perform a restore :-(

[ ? ?0.176269] omap_gpio_runtime_resume: bank @ 0xfc310000 [ ? ?0.177276] omap_gpio_runtime_resume: count 0, now 1
[ ? ?0.177276] gpiochip_add: registered GPIOs 0 to 31 on device: gpio [ ? ?0.177642] omap_gpio_runtime_suspend: bank @ 0xfc310000

Can you try ...

drivers/gpio/gpio-omap.c +0 -0

index c4ed172..9623408 100644
@@ -1086,6 +1086,9 @@ static int __devinit omap_gpio_probe(struct
 platform_device *pdev)

?#ifdef CONFIG_OF_GPIO
? ? ? ? bank->chip.of_node = of_node_get(node);
?#endif
+ ? ? ? if (bank->get_context_loss_count)
+ ? ? ? ? ? ? ? bank->context_loss_count =
+ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? bank->get_context_loss_count(bank->dev);

Quoted 3 lines

? ? ? ? bank->irq_base = irq_alloc_descs(-1, 0, bank->width, 0); ? ? ? ? if (bank->irq_base < 0) {

Looks like you found the culprit. :) It does fix the problem.

Franky

Quoted 28 lines · Franky Lin

Quoted 27 lines · Jon Hunter

I am wondering if this could be the bug ... on start-up I see that we do a context restore on bank1 during the probe which is before we have done the first suspend! In other words, we could restore a bad/uninitialised context for bank1. In the case of bank1, the loss count starts at 1 and not 0 and so we falsely think we need to perform a restore :-(

[    0.176269] omap_gpio_runtime_resume: bank @ 0xfc310000
[    0.177276] omap_gpio_runtime_resume: count 0, now 1
[    0.177276] gpiochip_add: registered GPIOs 0 to 31 on device: gpio
[    0.177642] omap_gpio_runtime_suspend: bank @ 0xfc310000

Can you try ...

drivers/gpio/gpio-omap.c +3 -0

index c4ed172..9623408 100644
@@ -1086,6 +1086,9 @@ static int __devinit omap_gpio_probe(struct
 platform_device *pdev)
 #ifdef CONFIG_OF_GPIO
        bank->chip.of_node = of_node_get(node);
 #endif
+       if (bank->get_context_loss_count)
+               bank->context_loss_count =
+                               bank->get_context_loss_count(bank->dev);
 

Quoted 5 lines

Quoted 3 lines

drivers/gpio/gpio-omap.c +0 -0

        bank->irq_base = irq_alloc_descs(-1, 0, bank->width, 0);

        if (bank->irq_base < 0) {

Looks like you found the culprit. :) It does fix the problem.

So this looks similar to what NeilBrown <[email protected]> reported in another thread.
The reason was context_loss_count = 1 for GPIO BANK#0 which of course is in the WKUP domain. In fact he tried out with the same fix. Anyways, we should hear from
Kevin now whether it is feasible to fix the context_loss_count for the WKUP GPIO bank or to put the workaround here in the gpio driver.

Hi Kevin, Tarun,

We are using the expansion connector A on Panda board to mount a SDIO WiFi dongle on MMC2 with a level triggered interrupt signal connected to GPIO 138. It's been working fine until 3.5 rc1. The board hang randomly within 5 mins during a network traffic test. After bisecting we found the culprit is "[PATCH 8/8] gpio/omap: fix missing check in *_runtime_suspend()" [1].

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

Hi Franky,

Quoted 8 lines · Franky Lin

Hi Kevin, Tarun,

We are using the expansion connector A on Panda board to mount a SDIO WiFi dongle on MMC2 with a level triggered interrupt signal connected to GPIO 138. It's been working fine until 3.5 rc1. The board hang randomly within 5 mins during a network traffic test. After bisecting we found the culprit is "[PATCH 8/8] gpio/omap: fix missing check in *_runtime_suspend()" [1].

I have been looking into this today to see if I can replicate the problem that you have reported. However, so far I have not had any luck. Please note that my test setup is not exactly the same as yours as I don't have your wlan module. However, I have been using a 2nd board to generate gpio events to a panda-es to see I can make it lock up. I have tried mainline kernel 3.5-rc1 and 3.5-rc3 but I have not seen any problems after sending 100k gpio events (over many minutes). My setup is as follows ...

- OMAP4460 panda-es with gpio-138 connected to OMAP3430 beagle gpio-11.
- Mainline kernel 3.5-rc1/3 using omap2plus_defconfig (no changes)
- Created a simple kernel module that acquires gpio-138 and sets up a IRQ with flag IRQF_TRIGGER_HIGH (for active high level interrupt).
- GPIO events are triggered roughly every 1ms

Can you confirm ...
1. You are just using omap2plus_defconfig with no changes?

2. Rough frequency of gpio events?

3. Is the gpio configured for active low or high?

4. When the hang occurs, what is the state of the gpio? Active or inactive? Can you probe it with a scope? If it was always active I could see that this would lock the device up, but I am not sure how that would relate to the results from your bisect???

Quoted 6 lines

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

It does not make sense to me yet why this would only impact 4460, but I will keep this in mind.

In your wlan driver are you acquiring and freeing the gpio often? Or are you only acquiring the gpio on boot?

The reason I ask is because for omap4, it seems that we are not currently calling omap2_gpio_prepare_for_idle() during idle and so the only time I see us call the runtime_suspend/resume handlers for omap4 is during probe and when we acquire and free the gpio.

So if you were not acquiring and freeing the gpio and are using the stock kernel, then as far as I can tell, the runtime pm code is not being exercised much. My test is not acquiring and releasing the gpio and so I am wondering if that is the secret to reproducing this problem :-)

Quoted 21 lines · Jon Hunter

Hi Franky,

Quoted 8 lines · Franky Lin

Hi Kevin, Tarun,

We are using the expansion connector A on Panda board to mount a SDIO WiFi dongle on MMC2 with a level triggered interrupt signal connected to GPIO 138. It's been working fine until 3.5 rc1. The board hang randomly within 5 mins during a network traffic test. After bisecting we found the culprit is "[PATCH 8/8] gpio/omap: fix missing check in *_runtime_suspend()" [1].

I have been looking into this today to see if I can replicate the problem that you have reported. However, so far I have not had any luck. Please note that my test setup is not exactly the same as yours as I don't have your wlan module. However, I have been using a 2nd board to generate gpio events to a panda-es to see I can make it lock up. I have tried mainline kernel 3.5-rc1 and 3.5-rc3 but I have not seen any problems after sending 100k gpio events (over many minutes). My setup is as follows ...

Quoted 5 lines

- OMAP4460 panda-es with gpio-138 connected to OMAP3430 beagle gpio-11.
- Mainline kernel 3.5-rc1/3 using omap2plus_defconfig (no changes)
- Created a simple kernel module that acquires gpio-138 and sets up a IRQ with flag IRQF_TRIGGER_HIGH (for active high level interrupt).
- GPIO events are triggered roughly every 1ms

Don't know if it's related, but we also mux several other pins on connector A:
/* MMC2 Mux for extension board */
/* MMC2 CMD */
OMAP4_MUX(GPMC_NWE, OMAP_MUX_MODE1 | OMAP_PIN_INPUT_PULLUP), /* MMC2 CLK */
OMAP4_MUX(GPMC_NOE, OMAP_MUX_MODE1 | OMAP_PIN_INPUT_PULLUP), /* MMC2 DAT 0-3 */
OMAP4_MUX(GPMC_AD0, OMAP_MUX_MODE1 | OMAP_PIN_INPUT_PULLUP), OMAP4_MUX(GPMC_AD1, OMAP_MUX_MODE1 | OMAP_PIN_INPUT_PULLUP), OMAP4_MUX(GPMC_AD2, OMAP_MUX_MODE1 | OMAP_PIN_INPUT_PULLUP), OMAP4_MUX(GPMC_AD3, OMAP_MUX_MODE1 | OMAP_PIN_INPUT_PULLUP), /* GPIO MUX for OOB interupt of dongle */ OMAP4_MUX(MCSPI1_CS1, OMAP_MUX_MODE3 | OMAP_PIN_INPUT_PULLDOWN), /* GPIO MUX for WLAN_ENABLE for dongle */ OMAP4_MUX(MCSPI1_CLK, OMAP_MUX_MODE3 | OMAP_PIN_OUTPUT),

Quoted 2 lines

Can you confirm ...
1. You are just using omap2plus_defconfig with no changes?

No, we enable following options
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_USB_OHCI_HCD=y

Quoted 1 line

2. Rough frequency of gpio events?

3367 interrupts were triggered during a 10 secs throughput test.

Quoted 1 line

3. Is the gpio configured for active low or high?

active high

Quoted 4 lines

4. When the hang occurs, what is the state of the gpio? Active or inactive? Can you probe it with a scope? If it was always active I could see that this would lock the device up, but I am not sure how that would relate to the results from your bisect???

I dont have a scope nearby. Let me see if I can find one tomorrow.

Quoted 18 lines

Quoted 6 lines

I noticed Kevin raised some similar cases on other platforms and also provided two patches in the patch mail thread. But unfortunately those two patches doesn't help in our case. I tested the driver with 3.5-rc3 mainline kernel and the issue is still there. I can only "fix" the hang by either reverting the commit or disabling CONFIG_PM_RUNTIME. Also, the hang only happens on Panda ES board. Old Panda with 4430 works good.

It does not make sense to me yet why this would only impact 4460, but I will keep this in mind.

In your wlan driver are you acquiring and freeing the gpio often? Or are you only acquiring the gpio on boot?

The reason I ask is because for omap4, it seems that we are not currently calling omap2_gpio_prepare_for_idle() during idle and so the only time I see us call the runtime_suspend/resume handlers for omap4 is during probe and when we acquire and free the gpio.

Quoted 5 lines

So if you were not acquiring and freeing the gpio and are using the stock kernel, then as far as I can tell, the runtime pm code is not being exercised much. My test is not acquiring and releasing the gpio and so I am wondering if that is the secret to reproducing this problem :-)

We only request the irq once during initialization. But we do frequently disable and re-enable it since we need to access to the module through SDIO to clear the interrupt. Apparently we can't finish all this in irq handler.

Hope these could help.