Failed to wake device (9984)
This is with our hacked 4.9 kernel, hacked firmware, and hacked driver (yah me!).
Sometimes on reboots, in at least some systems, the device is not woken up correctly, and of course it never recovers.
Any idea if there is some way to retry this a few times to see if the device can be recovered?
1177 Sep 14 17:00:09 ct524 kernel: ath10k driver, optimized for CT firmware, probing pci device: 0x46. 1178 Sep 14 17:00:09 ct524 systemd[1]: Reached target Swap.
1179 Sep 14 17:00:09 ct524 kernel: ath10k_pci 0000:04:00.0: failed to wake up device : -110 1180 Sep 14 17:00:09 ct524 kernel: ath10k_pci: probe of 0000:04:00.0 failed with error -110
Thanks,
Ben
Signature · 8 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
A message that is not in the archive (sent off-list or not yet received): CAJ-VmoknefqtxZCMc+PETo7Y3-HOcn0H3ySZqVWDTwMirACfcg@mail.gmail.com
Quoted 6 lines · Adrian Chadd
On 14 September 2017 at 17:08, Ben Greear <[email protected] <mailto:[email protected]>> wrote:
This is with our hacked 4.9 kernel, hacked firmware, and hacked driver (yah me!).
Quoted 8 lines
Sometimes on reboots, in at least some systems, the device is not woken up correctly, and of course it never recovers.
Any idea if there is some way to retry this a few times to see if the device can be recovered?
There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?
Can the cold reset be done on generic x86-64 hardware?
And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.
Thanks,
Ben
Quoted 25 lines
-adrian
1177 Sep 14 17:00:09 ct524 kernel: ath10k driver, optimized for CT firmware, probing pci device: 0x46. 1178 Sep 14 17:00:09 ct524 systemd[1]: Reached target Swap. 1179 Sep 14 17:00:09 ct524 kernel: ath10k_pci 0000:04:00.0: failed to wake up device : -110 1180 Sep 14 17:00:09 ct524 kernel: ath10k_pci: probe of 0000:04:00.0 failed with error -110 Thanks, Ben -- Ben Greear <[email protected] <mailto:[email protected]>> Candela Technologies Inc http://www.candelatech.com _______________________________________________ ath10k mailing list [email protected] <mailto:[email protected]> http://lists.infradead.org/mailman/listinfo/ath10k <http://lists.infradead.org/mailman/listinfo/ath10k>
Signature · 8 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
Quoted 5 lines · Ben Greear
Quoted 2 lines
There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?
Can the cold reset be done on generic x86-64 hardware?
I'll have to go check. You /should/ be able to. Are there are power and reset files in /sys/bus/pci for those devices?
Quoted 4 lines
And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.
Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back 0xfffffff or 0xdeadc0de or something?
-adrian
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
Quoted 12 lines · Adrian Chadd
Quoted 5 lines · Ben Greear
Quoted 2 lines
There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?
Can the cold reset be done on generic x86-64 hardware?
I'll have to go check. You /should/ be able to. Are there are power and reset files in
/sys/bus/pcifor those devices?Quoted 1 line
Quoted 8 lines
Quoted 3 lines
And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.
Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back
0xfffffffor0xdeadc0deor something?
One thing I notice, if I simply: rmmod ath10k_pci ath10k_core; modprobe ath10k_pci then it recovered (1 of 1 so far).
We'll see if that is a reliable way to recover from this problem. And, will see if we can also find a nicer way to go about it...maybe there is just a timer that is not long enough somewhere?
Thanks,
Ben
Signature · 8 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
Quoted 29 lines · Ben Greear
Quoted 22 lines · Adrian Chadd
Quoted 8 lines · Ben Greear
Quoted 4 lines
There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?
Can the cold reset be done on generic x86-64 hardware?
I'll have to go check. You /should/ be able to. Are there are power and reset files in
/sys/bus/pcifor those devices?Quoted 4 lines
And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.
Quoted 4 lines
Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back
0xfffffffor0xdeadc0deor something?
Quoted 3 lines
One thing I notice, if I simply: rmmod
ath10k_pciath10k_core; modprobeath10k_pci
then it recovered (1 of 1 so far).
See if that's reliable. For QCA9880 I know it needed a full reacharound sometimes (ie, the reference driver has hooks to reach back into the PCIe nexus to toggle reset.)
Quoted 5 lines
We'll see if that is a reliable way to recover from this problem. And, will see if we
can also find a nicer way to go about it...maybe there is just a timer that is not long
enough somewhere?
It's possible. I am just always wary about their host glue in the chip :-) If reloading the driver helps then great. But all that /should/ be dong is a cold reset / wakeup..
-adrian
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
Quoted 30 lines · Adrian Chadd
Quoted 29 lines · Ben Greear
Quoted 26 lines · Adrian Chadd
Quoted 8 lines · Ben Greear
Quoted 4 lines
There were always weird cold reset races that necessitated a PCI bus reset of the device. :( can you even see the device? do any of the registers work?
Can the cold reset be done on generic x86-64 hardware?
I'll have to go check. You /should/ be able to. Are there are power and reset files in
/sys/bus/pcifor those devices?Quoted 4 lines
And, it shows up enough that the system probes it, at least. I guess no infrastructure to speak of set up for this thing, so not sure how to probe any registers.
Well, that could be cached BAR information. There are some cold / warm reset registers in the RTC block that are used during initial wakeup; print what they're saying to see if it's coming back
0xfffffffor0xdeadc0deor something?
Quoted 7 lines
Quoted 3 lines
One thing I notice, if I simply: rmmod
ath10k_pciath10k_core; modprobeath10k_pci
then it recovered (1 of 1 so far).See if that's reliable. For QCA9880 I know it needed a full reacharound sometimes (ie, the reference driver has hooks to reach back into the PCIe nexus to toggle reset.)
It is not that reliable. I'm now trying a hack to re-probe the bus up to 3 times if we fail....hoping maybe that will help.
We just hit a case where the first 2 times failed, but it booted on the third.
My patch looks like this:
drivers/net/wireless/ath/ath10k/pci.c +18 -2
@@ -3492,8 +3492,8 @@ static const struct ath10k_bus_ops ath10k_pci_bus_ops = { .get_num_banks = ath10k_pci_get_num_banks, }; -static int ath10k_pci_probe (struct pci_dev *pdev, -const struct pci_device_id *pci_dev) +static int __ath10k_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *pci_dev) {int ret =0 ;struct ath10k *ar;
@@ -3668,6 +3668,22 @@ static int ath10k_pci_probe(struct pci_dev *pdev,return ret; } +static int ath10k_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *pci_dev) +{ +int cnt =0 ; +int rv; +do { + rv =__ath10k_pci_probe (pdev, pci_dev); +if (rv ==0 ) +return rv; +pr_err ("ath10k: failed to probe PCI : %d, retry-count: %d\n" , rv, cnt); +udelay (10000 );/* let the ath10k firmware gerbil take a small break */ + }while (cnt++ <3 ); +return rv; +} + +static void ath10k_pci_remove (struct pci_dev *pdev) {struct ath10k *ar =pci_get_drvdata (pdev);
Thanks,
Ben
Quoted 20 lines
Quoted 6 lines
We'll see if that is a reliable way to recover from this problem. And, will see if we
can also find a nicer way to go about it...maybe there is just a timer that is not long
enough somewhere?It's possible. I am just always wary about their host glue in the chip :-) If reloading the driver helps then great. But all that /should/ be dong is a cold reset / wakeup..
-adrian
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
Signature · 8 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
Quoted 5 lines · Ben Greear
It is not that reliable. I'm now trying a hack to re-probe the bus up to 3 times if we fail....hoping maybe that will help.
We just hit a case where the first 2 times failed, but it booted on the third.
See if there's a call to reset and reprobe/re-BAR the PCI resources that you can do from inside of linux kernel code.
-adrian
Quoted 75 lines
My patch looks like this:
drivers/net/wireless/ath/ath10k/pci.c +2
-2@@ -3492,8 +3492,8 @@ static const struct ath10k_bus_ops ath10k_pci_bus_ops = { .get_num_banks = ath10k_pci_get_num_banks, }; -static int ath10k_pci_probe (struct pci_dev *pdev, -const struct pci_device_id *pci_dev) +static int __ath10k_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *pci_dev) {int ret =0 ;struct ath10k *ar;drivers/net/wireless/ath/ath10k/pci.c +11
-0@@ -3668,6 +3668,22 @@ static int ath10k_pci_probe(struct pci_dev *pdev,return ret; } +static int ath10k_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *pci_dev) +{ +int cnt =0 ; +int rv; + do { + rv = __ath10k_pci_probe(pdev, pci_dev); + if (rv == 0) + return rv; + pr_err("ath10k: failed to probe PCI : %d, retry-count: %d\n", rv, cnt); + udelay(10000); /* let the ath10k firmware gerbil take asmall break */
drivers/net/wireless/ath/ath10k/pci.c +5
-0+ }while (cnt++ <3 ); +return rv; +} + +static void ath10k_pci_remove (struct pci_dev *pdev) {struct ath10k *ar =pci_get_drvdata (pdev);Thanks,
BenQuoted 16 lines
Quoted 8 lines
We'll see if that is a reliable way to recover from this problem. And, will
see if we
can also find a nicer way to go about it...maybe there is just a timer that
is not long
enough somewhere?It's possible. I am just always wary about their host glue in the chip :-) If reloading the driver helps then great. But all that /should/ be dong is a cold reset / wakeup..
Quoted 7 lines
-adrian
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
Quoted 4 lines
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k