Skip to contentSkip to conversation

More issues with ath10k_flush

DISCUSSION Ben Greear 2 participants 4 messages Jun 5, 2014 – Jun 6, 2014 Qualcomm Atheros ATH10K wireless driver

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

I'm back to debugging this charmer.

Currently I see the flush fail (and take 5 seconds doing so) fairly often when creating lots of station vifs against my firmware.

Once stations are connected, there are usually no more timeouts, even though I might be sending/receiving 100+Mbps of traffic for hours at a time.

By printing out the firmware stats, I see that much of the time the hardware has accepted X packets for transmission, but has completed X-1. It is possible the firmware's counters are screwed up some how or that it lost a packet, but I think it may also be possible that the firmware is just being really slow about completing a packet every now and then. I have looked at the firmware in detail and have found no way that it could actually leak tx descriptors.

So, I was thinking about changing the flush logic to try the current flush (that just waits) for up to 1/5 of the flush timeout, and if that fails, try telling the firmware to purge it's tx buffers, and then wait up to 4/5ths more of the flush timeout.

Does that sound like a reasonable approach?

Currently, my work-around is just to restart firmware after it fails to flush for 2 tries in a row, seems like there could be something better!

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#1 lore raw
Quoted 17 lines · Ben Greear

I'm back to debugging this charmer.

Currently I see the flush fail (and take 5 seconds doing so) fairly often when creating lots of station vifs against my firmware.

Once stations are connected, there are usually no more timeouts, even though I might be sending/receiving 100+Mbps of traffic for hours at a time.

By printing out the firmware stats, I see that much of the time the hardware has accepted X packets for transmission, but has completed X-1. It is possible the firmware's counters are screwed up some how or that it lost a packet, but I think it may also be possible that the firmware is just being really slow about completing a packet every now and then. I have looked at the firmware in detail and have found no way that it could actually leak tx descriptors.

Quoted 5 lines

So, I was thinking about changing the flush logic to try the current flush (that just waits) for up to 1/5 of the flush timeout, and if that fails, try telling the firmware to purge it's tx buffers, and then wait up to 4/5ths more of the flush timeout.

After poking around, it seems there is no wmi command to tell the firmware to just flush everything, so I hacked one into my firmware, called it before ath10k_flush starts waiting, and after several reboots, I do not see any timeouts trying to flush.

So, maybe that will do the trick...other suggestions are still welcome :)

Ben

Quoted 10 lines

Does that sound like a reasonable approach?

Currently, my work-around is just to restart firmware after it fails to flush for 2 tries in a row, seems like there could be something better!

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#2 lore raw
Quoted 10 lines · Ben Greear
Quoted 9 lines · Ben Greear

I'm back to debugging this charmer.

Currently I see the flush fail (and take 5 seconds doing so) fairly often when creating lots of station vifs against my firmware.

Once stations are connected, there are usually no more timeouts, even though I might be sending/receiving 100+Mbps of traffic for hours at a time.

Quoted 7 lines
Quoted 7 lines

By printing out the firmware stats, I see that much of the time the hardware has accepted X packets for transmission, but has completed X-1. It is possible the firmware's counters are screwed up some how or that it lost a packet, but I think it may also be possible that the firmware is just being really slow about completing a packet every now and then. I have looked at the firmware in detail and have found no way that it could actually leak tx descriptors.

Interesting. This reminds me the lazy wmi-htc tx credit replenishment after wmi mgmt tx is completed. Maybe it's a similar sort of thing? Maybe it's actually completed but for some reason the completion hasn't been fully processed yet..

Quoted 5 lines
Quoted 5 lines

So, I was thinking about changing the flush logic to try the current flush (that just waits) for up to 1/5 of the flush timeout, and if that fails, try telling the firmware to purge it's tx buffers, and then wait up to 4/5ths more of the flush timeout.

Sounds reasonable.

Quoted 5 lines

After poking around, it seems there is no wmi command to tell the firmware to just flush everything, so I hacked one into my firmware, called it before ath10k_flush starts waiting, and after several reboots, I do not see any timeouts trying to flush.

I thought WMI_PEER_FLUSH_TIDS_CMDID is for that. It didn't work for you? If so I would assume it's a firmware bug..

Quoted 2 lines

So, maybe that will do the trick...other suggestions are still welcome :)

Did you try to find out what kind of frame is supposedly held? I recall you've posted a NullFunc hexdump once pointing that it's one of the offending frames that didn't complete.

So.. maybe just not sending NullFunc frames (hell, they don't get a proper ack status anyway..) or somehow altering how they are sent is another way to work this around.

Michał

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#3 lore raw
Quoted 19 lines · Michal Kazior
Quoted 10 lines · Ben Greear
Quoted 9 lines · Ben Greear

I'm back to debugging this charmer.

Currently I see the flush fail (and take 5 seconds doing so) fairly often when creating lots of station vifs against my firmware.

Once stations are connected, there are usually no more timeouts, even though I might be sending/receiving 100+Mbps of traffic for hours at a time.

Quoted 7 lines
Quoted 7 lines

By printing out the firmware stats, I see that much of the time the hardware has accepted X packets for transmission, but has completed X-1. It is possible the firmware's counters are screwed up some how or that it lost a packet, but I think it may also be possible that the firmware is just being really slow about completing a packet every now and then. I have looked at the firmware in detail and have found no way that it could actually leak tx descriptors.

Quoted 4 lines

Interesting. This reminds me the lazy wmi-htc tx credit replenishment after wmi mgmt tx is completed. Maybe it's a similar sort of thing? Maybe it's actually completed but for some reason the completion hasn't been fully processed yet..

I didn't see any reason for that to happen in the firmware, but it is not the simplest code...

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

So, I was thinking about changing the flush logic to try the current flush (that just waits) for up to 1/5 of the flush timeout, and if that fails, try telling the firmware to purge it's tx buffers, and then wait up to 4/5ths more of the flush timeout.

Sounds reasonable.

By flushing before we start waiting, maybe we don't need the extra cleverness...but possibly it would be better to wait a short bit of time an then flush firmware if we still have pending skbs?

Quoted 8 lines
Quoted 5 lines

After poking around, it seems there is no wmi command to tell the firmware to just flush everything, so I hacked one into my firmware, called it before ath10k_flush starts waiting, and after several reboots, I do not see any timeouts trying to flush.

I thought WMI_PEER_FLUSH_TIDS_CMDID is for that. It didn't work for you? If so I would assume it's a firmware bug..

Well, actually, the command may have worked...but instead of iterating through all peers for all vdevs and making lots of wmi calls, I just made the firmware do the iteration by passing 0xFFFFFFFF as the vdev-id and special-casing the firmware handling of the message.

Was only about 8 extra lines of code in the firmware...

I also noticed something where the firmware might not be flushing it's tids when a vdev goes down...I didn't bother to change that yet, but possibly that is part of the issue. (It only flushed if vdev was 'paused'...not sure why.)

Quoted 10 lines
Quoted 2 lines

So, maybe that will do the trick...other suggestions are still welcome :)

Did you try to find out what kind of frame is supposedly held? I recall you've posted a NullFunc hexdump once pointing that it's one of the offending frames that didn't complete.

So.. maybe just not sending NullFunc frames (hell, they don't get a proper ack status anyway..) or somehow altering how they are sent is another way to work this around.

I haven't tried printing them lately...and if the flush logic continues to work, I probably won't bother...

In the past, I know there were sometimes lots of larger frames as well, but possibly that was a separate issue as I have not seen more than one frame hung lately.

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k