Skip to contentSkip to conversation

Failed to wake device (9984)

DISCUSSION Ben Greear 2 participants 7 messages Sep 15, 2017 1 message not in the archive Qualcomm Atheros ATH10K wireless driver

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

This is with our hacked 4.9 kernel, hacked firmware, and hacked driver (yah me!).

Sometimes on reboots, in at least some systems, the device is not woken up correctly, and of course it never recovers.

Any idea if there is some way to retry this a few times to see if the device can be recovered?

1177 Sep 14 17:00:09 ct524 kernel: ath10k driver, optimized for CT firmware, probing pci device: 0x46. 1178 Sep 14 17:00:09 ct524 systemd[1]: Reached target Swap.
1179 Sep 14 17:00:09 ct524 kernel: ath10k_pci 0000:04:00.0: failed to wake up device : -110 1180 Sep 14 17:00:09 ct524 kernel: ath10k_pci: probe of 0000:04:00.0 failed with error -110

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

A message that is not in the archive (sent off-list or not yet received): CAJ-VmoknefqtxZCMc+PETo7Y3-HOcn0H3ySZqVWDTwMirACfcg@mail.gmail.com

author
#2 lore raw
Quoted 6 lines · Adrian Chadd

On 14 September 2017 at 17:08, Ben Greear <[email protected] <mailto:[email protected]>> wrote:

This is with our hacked 4.9 kernel, hacked firmware, and hacked driver (yah me!).

Quoted 8 lines

Sometimes on reboots, in at least some systems, the device is not woken up correctly, and of course it never recovers.

Any idea if there is some way to retry this a few times to see if the device can be recovered?

There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?

Can the cold reset be done on generic x86-64 hardware?

And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.

Thanks,
Ben

Quoted 25 lines

-adrian

    1177 Sep 14 17:00:09 ct524 kernel: ath10k driver, optimized for CT firmware, probing pci device: 0x46.
    1178 Sep 14 17:00:09 ct524 systemd[1]: Reached target Swap.
    1179 Sep 14 17:00:09 ct524 kernel: ath10k_pci 0000:04:00.0: failed to wake up device : -110
    1180 Sep 14 17:00:09 ct524 kernel: ath10k_pci: probe of 0000:04:00.0 failed with error -110


    Thanks,
    Ben
    --
    Ben Greear <[email protected] <mailto:[email protected]>>
    Candela Technologies Inc  http://www.candelatech.com


    _______________________________________________
    ath10k mailing list
    [email protected] <mailto:[email protected]>
    http://lists.infradead.org/mailman/listinfo/ath10k <http://lists.infradead.org/mailman/listinfo/ath10k>
Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#3 lore raw
Quoted 5 lines · Ben Greear
Quoted 2 lines

There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?

Can the cold reset be done on generic x86-64 hardware?

I'll have to go check. You /should/ be able to. Are there are power and reset files in /sys/bus/pci for those devices?

Quoted 4 lines

And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.

Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back 0xfffffff or 0xdeadc0de or something?

-adrian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#4 lore raw
Quoted 12 lines · Adrian Chadd
Quoted 5 lines · Ben Greear
Quoted 2 lines

There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?

Can the cold reset be done on generic x86-64 hardware?

I'll have to go check. You /should/ be able to. Are there are power and reset files in /sys/bus/pci for those devices?

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.

Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back 0xfffffff or 0xdeadc0de or something?

One thing I notice, if I simply: rmmod ath10k_pci ath10k_core; modprobe ath10k_pci then it recovered (1 of 1 so far).

We'll see if that is a reliable way to recover from this problem. And, will see if we can also find a nicer way to go about it...maybe there is just a timer that is not long enough somewhere?

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#5 lore raw
Quoted 29 lines · Ben Greear
Quoted 22 lines · Adrian Chadd
Quoted 8 lines · Ben Greear
Quoted 4 lines

There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?

Can the cold reset be done on generic x86-64 hardware?

I'll have to go check. You /should/ be able to. Are there are power and reset files in /sys/bus/pci for those devices?

Quoted 4 lines

And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.

Quoted 4 lines

Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back 0xfffffff or 0xdeadc0de or something?

Quoted 3 lines

One thing I notice, if I simply: rmmod ath10k_pci ath10k_core; modprobe ath10k_pci
then it recovered (1 of 1 so far).

See if that's reliable. For QCA9880 I know it needed a full reacharound sometimes (ie, the reference driver has hooks to reach back into the PCIe nexus to toggle reset.)

Quoted 5 lines

We'll see if that is a reliable way to recover from this problem. And, will see if we
can also find a nicer way to go about it...maybe there is just a timer that is not long
enough somewhere?

It's possible. I am just always wary about their host glue in the chip :-) If reloading the driver helps then great. But all that /should/ be dong is a cold reset / wakeup..

-adrian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#6 lore raw
Quoted 30 lines · Adrian Chadd
Quoted 29 lines · Ben Greear
Quoted 26 lines · Adrian Chadd
Quoted 8 lines · Ben Greear
Quoted 4 lines

There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?

Can the cold reset be done on generic x86-64 hardware?

I'll have to go check. You /should/ be able to. Are there are power and reset files in /sys/bus/pci for those devices?

Quoted 4 lines

And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.

Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back 0xfffffff or 0xdeadc0de or something?

Quoted 7 lines
Quoted 3 lines

One thing I notice, if I simply: rmmod ath10k_pci ath10k_core; modprobe ath10k_pci
then it recovered (1 of 1 so far).

See if that's reliable. For QCA9880 I know it needed a full reacharound sometimes (ie, the reference driver has hooks to reach back into the PCIe nexus to toggle reset.)

It is not that reliable. I'm now trying a hack to re-probe the bus up to 3 times if we fail....hoping maybe that will help.

We just hit a case where the first 2 times failed, but it booted on the third.

My patch looks like this:

drivers/net/wireless/ath/ath10k/pci.c +18 -2
index e0a7b338..711b3f0 100644
@@ -3492,8 +3492,8 @@ static const struct ath10k_bus_ops ath10k_pci_bus_ops = {
        .get_num_banks  = ath10k_pci_get_num_banks,
 };
 
-static int ath10k_pci_probe(struct pci_dev *pdev,
-                           const struct pci_device_id *pci_dev)
+static int __ath10k_pci_probe(struct pci_dev *pdev,
+                             const struct pci_device_id *pci_dev)
 {
        int ret = 0;
        struct ath10k *ar;
@@ -3668,6 +3668,22 @@ static int ath10k_pci_probe(struct pci_dev *pdev,
        return ret;
 }
 
+static int ath10k_pci_probe(struct pci_dev *pdev,
+                           const struct pci_device_id *pci_dev)
+{
+       int cnt = 0;
+       int rv;
+       do {
+               rv = __ath10k_pci_probe(pdev, pci_dev);
+               if (rv == 0)
+                       return rv;
+               pr_err("ath10k: failed to probe PCI : %d, retry-count: %d\n", rv, cnt);
+               udelay(10000); /* let the ath10k firmware gerbil take a small break */
+       } while (cnt++ < 3);
+       return rv;
+}
+
+
 static void ath10k_pci_remove(struct pci_dev *pdev)
 {
        struct ath10k *ar = pci_get_drvdata(pdev);

Thanks,
Ben

Quoted 20 lines
Quoted 6 lines

We'll see if that is a reliable way to recover from this problem. And, will see if we
can also find a nicer way to go about it...maybe there is just a timer that is not long
enough somewhere?

It's possible. I am just always wary about their host glue in the chip :-) If reloading the driver helps then great. But all that /should/ be dong is a cold reset / wakeup..

-adrian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#7 lore raw
Quoted 5 lines · Ben Greear

It is not that reliable. I'm now trying a hack to re-probe the bus up to 3 times if we fail....hoping maybe that will help.

We just hit a case where the first 2 times failed, but it booted on the third.

See if there's a call to reset and reprobe/re-BAR the PCI resources that you can do from inside of linux kernel code.

-adrian

Quoted 75 lines

My patch looks like this:

drivers/net/wireless/ath/ath10k/pci.c +2 -2
index e0a7b338..711b3f0 100644
@@ -3492,8 +3492,8 @@ static const struct ath10k_bus_ops ath10k_pci_bus_ops
 = {
        .get_num_banks  = ath10k_pci_get_num_banks,
 };
 
-static int ath10k_pci_probe(struct pci_dev *pdev,
-                           const struct pci_device_id *pci_dev)
+static int __ath10k_pci_probe(struct pci_dev *pdev,
+                             const struct pci_device_id *pci_dev)
 {
        int ret = 0;
        struct ath10k *ar;
drivers/net/wireless/ath/ath10k/pci.c +11 -0
@@ -3668,6 +3668,22 @@ static int ath10k_pci_probe(struct pci_dev *pdev,
        return ret;
 }
 
+static int ath10k_pci_probe(struct pci_dev *pdev,
+                           const struct pci_device_id *pci_dev)
+{
+       int cnt = 0;
+       int rv;
+       do {
+               rv = __ath10k_pci_probe(pdev, pci_dev);
+               if (rv == 0)
+                       return rv;
+               pr_err("ath10k: failed to probe PCI : %d, retry-count:
 %d\n", rv, cnt);
+               udelay(10000); /* let the ath10k firmware gerbil take a

small break */

drivers/net/wireless/ath/ath10k/pci.c +5 -0
+       } while (cnt++ < 3);
+       return rv;
+}
+
+
 static void ath10k_pci_remove(struct pci_dev *pdev)
 {
        struct ath10k *ar = pci_get_drvdata(pdev);

Thanks,
Ben

Quoted 16 lines
Quoted 8 lines

We'll see if that is a reliable way to recover from this problem. And, will
see if we
can also find a nicer way to go about it...maybe there is just a timer that
is not long
enough somewhere?

It's possible. I am just always wary about their host glue in the chip :-) If reloading the driver helps then great. But all that /should/ be dong is a cold reset / wakeup..

Quoted 7 lines

-adrian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

Quoted 4 lines

--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k