ath9k: Detect and work-around tx-queue hang.
RFC 1/1 ath9k: Detect and work-around tx-queue hang.
From: Ben Greear <[email protected]>
We see TX lockups on ar9380 NICs when running 32 stations each with a 56kbps stream of MTU sized UDP packets. We see lockups on the AP and also on the station, seems random which hits first.
The test case further involves a programmable attenuator, and the attenuation is taken from -30 to -85 signal level in steps of 10db. Each step runs for 1 minute before increasing the attenuation. The problem normally shows up around signal level of -70 (noise is reported as around -95).
When the lockup hits, it is typically on a single queue (BE). The symptom is that there is no obvious transmit activity on that queue, the acq-depth and axq-ampdu-depth are zero, the queue is stopped, and the pending-frames is at or above the maximum allowed. The VO queue continues to function, and RX logic functions fine.
Just resetting the chip does not fix the problem: The pending-frames usually stays at max. So, this patch also adds hacks to force pending-frames to zero. It also quietens some warnings about pending-frame underruns because sometimes, the tx status does appear many seconds later.
Finally, the reset fixup code is logged at ath_err because I think everyone should be aware of events like this.
We see the same problem with ath9k rate control and minstrel-ht. We have not tested other ath9k chipsets in this manner.
Small numbers of high-speed stations do not hit this problem, or at least not in our test cases.
--- drivers/net/wireless/ath/ath9k/ath9k.h | 2 ++ drivers/net/wireless/ath/ath9k/link.c | 30 ++++++++++++++++++++++++++++-- drivers/net/wireless/ath/ath9k/main.c | 5 +++-- drivers/net/wireless/ath/ath9k/xmit.c | 15 ++++++++++++++- 4 files changed, 47 insertions(+), 5 deletions(-)
drivers/net/wireless/ath/ath9k/ath9k.h +2 -0
@@ -194,6 +194,7 @@ struct ath_txq {u32 axq_ampdu_depth;bool stopped;bool axq_tx_inprogress; +bool clear_pending_frames_on_flush;struct list_head axq_acq;struct list_head txq_fifo[ATH_TXFIFO_DEPTH ];u8 txq_headidx;
@@ -684,6 +685,7 @@ struct ath_softc {u16 curtxpow;bool ps_enabled;bool ps_idle; +bool reset_force_noretry;short nbcnvifs;short nvifs;unsigned long ps_usecount;
drivers/net/wireless/ath/ath9k/link.c +28 -2
@@ -38,18 +38,44 @@ void ath_tx_complete_poll_work(struct work_struct *work)if (txq->axq_depth) {if (txq->axq_tx_inprogress) { needreset = true; +ath_err (ath9k_hw_common (sc->sc_ah), +"tx hung, queue: %i axq-depth: %i, ampdu-depth: %i resetting the chip\n" , + i, txq->axq_depth, + txq->axq_ampdu_depth);ath_txq_unlock (sc, txq);break ; }else { txq->axq_tx_inprogress = true; } + }else { +/* Check for software TX hang. It seems +* sometimes pending-frames is not properly +* decremented, and the tx queue hangs. +* Considered hung if: axq-depth is zero, +* ampdu-depth is zero, queue-is-stopped, +* and we have pending frames. +*/ +if (txq->stopped && + (txq->axq_ampdu_depth ==0 ) && + (txq->pending_frames >0 )) { +if (txq->axq_tx_inprogress) { +ath_err (ath9k_hw_common (sc->sc_ah), +"soft tx hang: queue: %i pending-frames: %i, resetting chip\n" , + i, txq->pending_frames); + needreset = true; + txq->clear_pending_frames_on_flush = true; + sc->reset_force_noretry = true; +ath_txq_unlock (sc, txq); +break ; + }else { + txq->axq_tx_inprogress = true; + } + } }ath_txq_unlock_complete (sc, txq); }if (needreset) { -ath_dbg (ath9k_hw_common (sc->sc_ah),RESET , -"tx hung, resetting the chip\n" );ath9k_queue_reset (sc,RESET_TYPE_TX_HANG );return ; }
drivers/net/wireless/ath/ath9k/main.c +3 -2
@@ -587,8 +587,9 @@ void ath9k_queue_reset(struct ath_softc *sc, enum ath_reset_type type)void ath_reset_work (struct work_struct *work) { struct ath_softc *sc = container_of(work, struct ath_softc, hw_reset_work); - -ath_reset (sc, true); +bool retry_tx = !sc->reset_force_noretry; + sc->reset_force_noretry = false; +ath_reset (sc, retry_tx); }/**********************/
drivers/net/wireless/ath/ath9k/xmit.c +14 -1
@@ -1543,6 +1543,15 @@ void ath_draintxq(struct ath_softc *sc, struct ath_txq *txq, bool retry_tx)if ((sc->sc_ah->caps.hw_caps &ATH9K_HW_CAP_HT ) && !retry_tx)ath_txq_drain_pending_buffers (sc, txq); +if (txq->clear_pending_frames_on_flush && (txq->pending_frames !=0 )) { +ath_err (ath9k_hw_common (sc->sc_ah), +"Pending frames still exist on txq: %i after drain: %i axq-depth: %i ampdu-depth: %i\n" , + txq->mac80211_qnum, txq->pending_frames, txq->axq_depth, + txq->axq_ampdu_depth); + txq->pending_frames =0 ; + } + txq->clear_pending_frames_on_flush = false; +ath_txq_unlock_complete (sc, txq); }
@@ -2066,8 +2075,12 @@ static void ath_tx_complete(struct ath_softc *sc, struct sk_buff *skb, q =skb_get_queue_mapping (skb);if (txq == sc->tx.txq_map[q]) { -if (WARN_ON (--txq->pending_frames <0 )) +if (--txq->pending_frames <0 ) { +if (net_ratelimit ()) +ath_err (common,"txq: %p had negative pending_frames, q: %i\n" , + txq, q); txq->pending_frames =0 ; + }if (txq->stopped && txq->pending_frames < sc->tx.txq_max_pending[q]) {
Signature · 1 line
1.7.3.4
Hi,
This is definitely a work-around. :)
I think we should debug a bit more to find out the actual bug rather than add more hacks to the already hackish TX poll routine.
Sujith
Quoted 174 lines · greearb at candelatech.com
From: Ben Greear <[email protected]>
We see TX lockups on ar9380 NICs when running 32 stations each with a 56kbps stream of MTU sized UDP packets. We see lockups on the AP and also on the station, seems random which hits first.
The test case further involves a programmable attenuator, and the attenuation is taken from -30 to -85 signal level in steps of 10db. Each step runs for 1 minute before increasing the attenuation. The problem normally shows up around signal level of -70 (noise is reported as around -95).
When the lockup hits, it is typically on a single queue (BE). The symptom is that there is no obvious transmit activity on that queue, the acq-depth and axq-ampdu-depth are zero, the queue is stopped, and the pending-frames is at or above the maximum allowed. The VO queue continues to function, and RX logic functions fine.
Just resetting the chip does not fix the problem: The pending-frames usually stays at max. So, this patch also adds hacks to force pending-frames to zero. It also quietens some warnings about pending-frame underruns because sometimes, the tx status does appear many seconds later.
Finally, the reset fixup code is logged at
ath_errbecause I think everyone should be aware of events like this.We see the same problem with ath9k rate control and minstrel-ht. We have not tested other ath9k chipsets in this manner.
Small numbers of high-speed stations do not hit this problem, or at least not in our test cases.
Signed-off-by Ben Greear <[email protected]>--- drivers/net/wireless/ath/ath9k/ath9k.h | 2 ++ drivers/net/wireless/ath/ath9k/link.c | 30 ++++++++++++++++++++++++++++-- drivers/net/wireless/ath/ath9k/main.c | 5 +++-- drivers/net/wireless/ath/ath9k/xmit.c | 15 ++++++++++++++- 4 files changed, 47 insertions(+), 5 deletions(-)drivers/net/wireless/ath/ath9k/ath9k.h +2
-0@@ -194,6 +194,7 @@ struct ath_txq {u32 axq_ampdu_depth;bool stopped;bool axq_tx_inprogress; +bool clear_pending_frames_on_flush;struct list_head axq_acq;struct list_head txq_fifo[ATH_TXFIFO_DEPTH ];u8 txq_headidx;@@ -684,6 +685,7 @@ struct ath_softc {u16 curtxpow;bool ps_enabled;bool ps_idle; +bool reset_force_noretry;short nbcnvifs;short nvifs;unsigned long ps_usecount;drivers/net/wireless/ath/ath9k/link.c +28
-2@@ -38,18 +38,44 @@ void ath_tx_complete_poll_work(struct work_struct *work)if (txq->axq_depth) {if (txq->axq_tx_inprogress) { needreset = true; +ath_err (ath9k_hw_common (sc->sc_ah), +"tx hung, queue: %i axq-depth: %i, ampdu-depth: %i resetting the chip\n" , + i, txq->axq_depth, + txq->axq_ampdu_depth);ath_txq_unlock (sc, txq);break ; }else { txq->axq_tx_inprogress = true; } + }else { +/* Check for software TX hang. It seems +* sometimes pending-frames is not properly +* decremented, and the tx queue hangs. +* Considered hung if: axq-depth is zero, +* ampdu-depth is zero, queue-is-stopped, +* and we have pending frames. +*/ +if (txq->stopped && + (txq->axq_ampdu_depth ==0 ) && + (txq->pending_frames >0 )) { +if (txq->axq_tx_inprogress) { +ath_err (ath9k_hw_common (sc->sc_ah), +"soft tx hang: queue: %i pending-frames: %i, resetting chip\n" , + i, txq->pending_frames); + needreset = true; + txq->clear_pending_frames_on_flush = true; + sc->reset_force_noretry = true; +ath_txq_unlock (sc, txq); +break ; + }else { + txq->axq_tx_inprogress = true; + } + } }ath_txq_unlock_complete (sc, txq); }if (needreset) { -ath_dbg (ath9k_hw_common (sc->sc_ah),RESET , -"tx hung, resetting the chip\n" );ath9k_queue_reset (sc,RESET_TYPE_TX_HANG );return ; }drivers/net/wireless/ath/ath9k/main.c +3
-2@@ -587,8 +587,9 @@ void ath9k_queue_reset(struct ath_softc *sc, enum ath_reset_type type)void ath_reset_work (struct work_struct *work) { struct ath_softc *sc = container_of(work, struct ath_softc, hw_reset_work); - -ath_reset (sc, true); +bool retry_tx = !sc->reset_force_noretry; + sc->reset_force_noretry = false; +ath_reset (sc, retry_tx); }/**********************/ drivers/net/wireless/ath/ath9k/xmit.c +14
-1@@ -1543,6 +1543,15 @@ void ath_draintxq(struct ath_softc *sc, struct ath_txq *txq, bool retry_tx)if ((sc->sc_ah->caps.hw_caps &ATH9K_HW_CAP_HT ) && !retry_tx)ath_txq_drain_pending_buffers (sc, txq); +if (txq->clear_pending_frames_on_flush && (txq->pending_frames !=0 )) { +ath_err (ath9k_hw_common (sc->sc_ah), +"Pending frames still exist on txq: %i after drain: %i axq-depth: %i ampdu-depth: %i\n" , + txq->mac80211_qnum, txq->pending_frames, txq->axq_depth, + txq->axq_ampdu_depth); + txq->pending_frames =0 ; + } + txq->clear_pending_frames_on_flush = false; +ath_txq_unlock_complete (sc, txq); }@@ -2066,8 +2075,12 @@ static void ath_tx_complete(struct ath_softc *sc, struct sk_buff *skb, q =skb_get_queue_mapping (skb);if (txq == sc->tx.txq_map[q]) { -if (WARN_ON (--txq->pending_frames <0 )) +if (--txq->pending_frames <0 ) { +if (net_ratelimit ()) +ath_err (common,"txq: %p had negative pending_frames, q: %i\n" , + txq, q); txq->pending_frames =0 ; + }if (txq->stopped && txq->pending_frames < sc->tx.txq_max_pending[q]) {Signature · 6 lines
1.7.3.4 -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Sujith Manoharan
Hi,
This is definitely a work-around. :)
I think we should debug a bit more to find out the actual bug rather than add more hacks to the already hackish TX poll routine.
I'll be happy to test patches, but I'm not sure how to go about debugging the real problem on my own. Maybe some stats could be added to the xmit debugfs file to help diagnose the problem, or maybe some other debugfs info will help?
I can't reproduce the problem with ath9k debugging set at the previous suggested level, so it would have to be something less invasive.
As for just stations going out of range, it remains locked up even with signal level goes back to -20, so it's not just a simple station-out-of range issues..
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 5 lines · Ben Greear
I'll be happy to test patches, but I'm not sure how to go about debugging the real problem on my own. Maybe some stats could be added to the xmit debugfs file to help diagnose the problem, or maybe some other debugfs info will help?
Quoted 7 lines
I can't reproduce the problem with ath9k debugging set at the previous suggested level, so it would have to be something less invasive.
As for just stations going out of range, it remains locked up even with signal level goes back to -20, so it's not just a simple station-out-of range issues..
Sure, but I think that filtered frames are not handled properly, especially with aggregation, since the debugfs stats from your earlier email showed a large counter (from a private patch ?):
TXERR Filtered: 224 0 0 0
Sujith
Quoted 14 lines · Sujith Manoharan
Quoted 5 lines · Ben Greear
I'll be happy to test patches, but I'm not sure how to go about debugging the real problem on my own. Maybe some stats could be added to the xmit debugfs file to help diagnose the problem, or maybe some other debugfs info will help?
Quoted 7 lines
I can't reproduce the problem with ath9k debugging set at the previous suggested level, so it would have to be something less invasive.
As for just stations going out of range, it remains locked up even with signal level goes back to -20, so it's not just a simple station-out-of range issues..
Quoted 5 lines
Sure, but I think that filtered frames are not handled properly, especially with aggregation, since the debugfs stats from your earlier email showed a large counter (from a private patch ?):
TXERR Filtered: 224 0 0 0
Yeah, guess that patch never made it upstream. The pertinent bit is:
+++ b/drivers/net/wireless/ath/ath9k/debug.c
@@ -579,6 +579,7 @@ static ssize_t read_file_xmit(struct file *file, char __user *user_buf,
PR("AMPDUs Completed:", a_completed);
PR("AMPDUs Retried: ", a_retries);
PR("AMPDUs XRetried: ", a_xretries);
+ PR("TXERR Filtered: ", txerr_filtered);
PR("FIFO Underrun: ", fifo_underrun);
PR("TXOP Exceeded: ", xtxop);
PR("TXTIMER Expiry: ", timer_exp);
@@ -867,6 +868,8 @@ void ath_debug_stat_tx(struct ath_softc *sc, struct ath_buf *bf,
TX_STAT_INC(qnum, completed);
}+ if (ts->ts_status & ATH9K_TXERR_FILT)
+ TX_STAT_INC(qnum, txerr_filtered);
if (ts->ts_status & ATH9K_TXERR_FIFO)
TX_STAT_INC(qnum, fifo_underrun);
if (ts->ts_status & ATH9K_TXERR_XTXOP)I'll post this and a few other small patches when I get a chance.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 15 lines · Ben Greear
Quoted 14 lines · Sujith Manoharan
Quoted 12 lines · Ben Greear
I'll be happy to test patches, but I'm not sure how to go about debugging the real problem on my own. Maybe some stats could be added to the xmit debugfs file to help diagnose the problem, or maybe some other debugfs info will help?
I can't reproduce the problem with ath9k debugging set at the previous suggested level, so it would have to be something less invasive.
As for just stations going out of range, it remains locked up even with signal level goes back to -20, so it's not just a simple station-out-of range issues..
Quoted 7 lines
Quoted 5 lines
Sure, but I think that filtered frames are not handled properly, especially with aggregation, since the debugfs stats from your earlier email showed a large counter (from a private patch ?):
TXERR Filtered: 224 0 0 0
Yeah, guess that patch never made it upstream. The pertinent bit is:
Please also check if the station(s) that the frames are queued for are in powersave state for some reason. That would prevent the tx path from throwing them in the hw queue, yet they'd still take up pending-frame slots. I was planning on fixing this eventually by expiring frames that stay in the queue for too long, but haven't decided on the exact approach yet.
- Felix
Quoted 6 lines · Felix Fietkau
Please also check if the station(s) that the frames are queued for are in powersave state for some reason. That would prevent the tx path from throwing them in the hw queue, yet they'd still take up pending-frame slots. I was planning on fixing this eventually by expiring frames that stay in the queue for too long, but haven't decided on the exact approach yet.
PS is disabled for multi-VIF.
Sujith
Quoted 9 lines · Sujith Manoharan
Quoted 6 lines · Felix Fietkau
Please also check if the station(s) that the frames are queued for are in powersave state for some reason. That would prevent the tx path from throwing them in the hw queue, yet they'd still take up pending-frame slots. I was planning on fixing this eventually by expiring frames that stay in the queue for too long, but haven't decided on the exact approach yet.
PS is disabled for multi-VIF.
What about off-channel PS due to scans, etc.
- Felix
Quoted 11 lines · Felix Fietkau
Quoted 9 lines · Sujith Manoharan
Quoted 6 lines · Felix Fietkau
Please also check if the station(s) that the frames are queued for are in powersave state for some reason. That would prevent the tx path from throwing them in the hw queue, yet they'd still take up pending-frame slots. I was planning on fixing this eventually by expiring frames that stay in the queue for too long, but haven't decided on the exact approach yet.
PS is disabled for multi-VIF.
What about off-channel PS due to scans, etc.
Scan is always locked to the same channel in this setup (once a single station is associated).
The stations stay associated while this problem happens (the high-priority queue seems to work just fine, which may be the reason they stay associated just fine.)
In some cases, I see packets delivered around 30 seconds late... aside from PS and off-channel..any idea what could make a packet stick around that long in the tx queues?
Thanks,
Ben
Quoted 3 lines
- Felix
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 16 lines · Ben Greear
Quoted 11 lines · Felix Fietkau
Quoted 9 lines · Sujith Manoharan
Quoted 6 lines · Felix Fietkau
Please also check if the station(s) that the frames are queued for are in powersave state for some reason. That would prevent the tx path from throwing them in the hw queue, yet they'd still take up pending-frame slots. I was planning on fixing this eventually by expiring frames that stay in the queue for too long, but haven't decided on the exact approach yet.
PS is disabled for multi-VIF.
What about off-channel PS due to scans, etc.
Scan is always locked to the same channel in this setup (once a single station is associated).
Quoted 7 lines
The stations stay associated while this problem happens (the high-priority queue seems to work just fine, which may be the reason they stay associated just fine.)
In some cases, I see packets delivered around 30 seconds late... aside from PS and off-channel..any idea what could make a packet stick around that long in the tx queues?
One of our customers on a 3.5.7+ kernel hit the problem without using any RF attenuator...just over-the-air communication to their AP. It happened on both 2.4 and 5Ghz bands. Seems rx signal is around 40 in their environment. It took them around 24 hours to hit the problem on average.
Last we checked, we could fairly easily reproduce this in our lab using an attenuator and a certain setup, so if there is any debugging we could add to help narrow down what might be causing this, we can give that a try.
For instance, is there any good way to know for certain if packets in the queue are in power-save or not? I know we at least attempt to disable power-save, but possibly it gets re-enabled somehow?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 4 lines · Ben Greear
For instance, is there any good way to know for certain if packets in the queue are in power-save or not? I know we at least attempt to disable power-save, but possibly it gets re-enabled somehow?
I am not sure if that could happen.
This issue got lost somehow, but reviewing xmit.c would probably give some clues. :)
Sujith
Quoted 36 lines · Ben Greear
Quoted 16 lines · Ben Greear
Quoted 11 lines · Felix Fietkau
Quoted 9 lines · Sujith Manoharan
Quoted 6 lines · Felix Fietkau
Please also check if the station(s) that the frames are queued for are in powersave state for some reason. That would prevent the tx path from throwing them in the hw queue, yet they'd still take up pending-frame slots. I was planning on fixing this eventually by expiring frames that stay in the queue for too long, but haven't decided on the exact approach yet.
PS is disabled for multi-VIF.
What about off-channel PS due to scans, etc.
Scan is always locked to the same channel in this setup (once a single station is associated).
Quoted 7 lines
The stations stay associated while this problem happens (the high-priority queue seems to work just fine, which may be the reason they stay associated just fine.)
In some cases, I see packets delivered around 30 seconds late... aside from PS and off-channel..any idea what could make a packet stick around that long in the tx queues?
One of our customers on a 3.5.7+ kernel hit the problem without using any RF attenuator...just over-the-air communication to their AP. It happened on both 2.4 and 5Ghz bands. Seems rx signal is around 40 in their environment. It took them around 24 hours to hit the problem on average.
Last we checked, we could fairly easily reproduce this in our lab using an attenuator and a certain setup, so if there is any debugging we could add to help narrow down what might be causing this, we can give that a try.
Quoted 4 lines
For instance, is there any good way to know for certain if packets in the queue are in power-save or not? I know we at least attempt to disable power-save, but possibly it gets re-enabled somehow?
The ath_node struct tracks if a node is sleeping or not. If a node is sleeping, its tid queues can still hold some frames but will not be serviced by ath_txq_schedule.
- Felix