Skip to contentSkip to conversation

rt2x00: rt2800usb: fix races in tx queue

PATCH Superseded Stanislaw Gruszka 3 participants 14 messages Aug 4, 2011 – Aug 10, 2011 Networking drivers (wireless)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rt2x00: rt2800usb: fix races in tx queue

We can perform parallel putting new entries in a queue (rt2x00queue_write_tx_frame()) and removing entries after finishing transmitting (rt2800usb_work_txdone()). There are cases when txdone may process an entry that was not fully send and nullify entry->skb. What in a result crash the kernel in rt2800usb_write_tx_desc() or rt2800usb_get_txwi() or other place where entry->skb is used.

To fix stop processing entries in txdone, which flags do not indicate transition was finished.

Also change flags as soon as we get confirmation transmision is done.

---
 drivers/net/wireless/rt2x00/rt2800usb.c |   36 ++++++++++++++++++++++--------
 drivers/net/wireless/rt2x00/rt2x00usb.c |    6 ++--
 2 files changed, 29 insertions(+), 13 deletions(-)
drivers/net/wireless/rt2x00/rt2800usb.c +26 -10
index 5075593..edd0518 100644
@@ -500,14 +500,14 @@ static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg)
 	return true;
 }
 
-static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
+static int rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 {
 	struct data_queue *queue;
 	struct queue_entry *entry;
 	u32 reg;
 	u8 qid;
 
-	while (kfifo_get(&rt2x00dev->txstatus_fifo, &reg)) {
+	while (kfifo_peek(&rt2x00dev->txstatus_fifo, &reg)) {
 
 		/* TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus
 		 * qid is guaranteed to be one of the TX QIDs
@@ -517,25 +517,39 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 		if (unlikely(!queue)) {
 			WARNING(rt2x00dev, "Got TX status for an unavailable "
 					   "queue %u, dropping\n", qid);
-			continue;
+			goto next_reg;
 		}
 
 		/*
 		 * Inside each queue, we process each entry in a chronological
 		 * order. We first check that the queue is not empty.
 		 */
-		entry = NULL;
-		while (!rt2x00queue_empty(queue)) {
+		while (1) {
+			entry = NULL;
+
+			if (rt2x00queue_empty(queue))
+				break;
+
 			entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE);
+
+			if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+			    !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+				WARNING(rt2x00dev, "Data pending for entry %u"
+					"in queue %u\n", entry->entry_idx, qid);
+				return 1;
+			}
+
 			if (rt2800usb_txdone_entry_check(entry, reg))
 				break;
 		}
 
-		if (!entry || rt2x00queue_empty(queue))
-			break;
-
-		rt2800_txdone_entry(entry, reg);
+		if (entry)
+			rt2800_txdone_entry(entry, reg);
+next_reg:
+		kfifo_skip(&rt2x00dev->txstatus_fifo);
 	}
+
+	return 0;
 }
 
 static void rt2800usb_work_txdone(struct work_struct *work)
@@ -545,7 +559,8 @@ static void rt2800usb_work_txdone(struct work_struct *work)
 	struct data_queue *queue;
 	struct queue_entry *entry;
 
-	rt2800usb_txdone(rt2x00dev);
+	if (rt2800usb_txdone(rt2x00dev))
+		goto out;
 
 	/*
 	 * Process any trailing TX status reports for IO failures,
@@ -569,6 +584,7 @@ static void rt2800usb_work_txdone(struct work_struct *work)
 		}
 	}
 
+out:
 	/*
 	 * The hw may delay sending the packet after DMA complete
 	 * if the medium is busy, thus the TX_STA_FIFO entry is
drivers/net/wireless/rt2x00/rt2x00usb.c +3 -3
index b6b4542..46c6d36 100644
@@ -265,14 +265,14 @@ static void rt2x00usb_interrupt_txdone(struct urb *urb)
 	if (!test_and_clear_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags))
 		return;
 
-	if (rt2x00dev->ops->lib->tx_dma_done)
-		rt2x00dev->ops->lib->tx_dma_done(entry);
-
 	/*
 	 * Report the frame as DMA done
 	 */
 	rt2x00lib_dmadone(entry);
 
+	if (rt2x00dev->ops->lib->tx_dma_done)
+		rt2x00dev->ops->lib->tx_dma_done(entry);
+
 	/*
 	 * Check if the frame was correctly uploaded
 	 */
Signature · 1 line
1.7.1
#1 lore raw

Hi,

Quoted 28 lines · Stanislaw Gruszka

We can perform parallel putting new entries in a queue (rt2x00queue_write_tx_frame()) and removing entries after finishing transmitting (rt2800usb_work_txdone()). There are cases when txdone may process an entry that was not fully send and nullify entry->skb. What in a result crash the kernel in rt2800usb_write_tx_desc() or rt2800usb_get_txwi() or other place where entry->skb is used.

To fix stop processing entries in txdone, which flags do not indicate transition was finished.

Also change flags as soon as we get confirmation transmision is done.

---
 drivers/net/wireless/rt2x00/rt2800usb.c |   36 ++++++++++++++++++++++-------- drivers/net/wireless/rt2x00/rt2x00usb.c |    6 ++--

 2 files changed, 29 insertions(+), 13 deletions(-)
drivers/net/wireless/rt2x00/rt2800usb.c +0 -0
index 5075593..edd0518 100644
@@ -500,14 +500,14 @@ static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg)
        return true;
  }
 

drivers/net/wireless/rt2x00/rt2800usb.c static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg) · line 510

Quoted 10 lines
drivers/net/wireless/rt2x00/rt2800usb.c +2 -2
-static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
+static int rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
  {
        struct data_queue *queue;
        struct queue_entry *entry;
        u32 reg;
        u8 qid;
 
-       while (kfifo_get(&rt2x00dev->txstatus_fifo, &reg)) {
+       while (kfifo_peek(&rt2x00dev->txstatus_fifo, &reg)) {

I'm not too sure about this change, why do you need to do kfifo_peek and add gotos to the end of the while-loop to remove the item from the queue? There is no condition in which the obtained value from kfifo-peek will require it to be read again later (because when the value couldn't be handled we are throwing it away anyway using kfifo_skip).

Ivo

Quoted 93 lines
drivers/net/wireless/rt2x00/rt2800usb.c +0 -0
                /* TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus
                 * qid is guaranteed to be one of the TX QIDs
@@ -517,25 +517,39 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
                if (unlikely(!queue)) {

                       WARNING(rt2x00dev, "Got TX status for an unavailable " "queue %u, dropping\n", qid);
-                       continue;
+                       goto next_reg;
               }

               /*
                * Inside each queue, we process each entry in a chronological
                * order. We first check that the queue is not empty. */
-               entry = NULL;
-               while (!rt2x00queue_empty(queue)) {
+               while (1) {
+                       entry = NULL;

drivers/net/wireless/rt2x00/rt2800usb.c +4 -0
+
+                       if (rt2x00queue_empty(queue))
+                               break;
+

                       entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE);

drivers/net/wireless/rt2x00/rt2800usb.c +8 -0
+
+                       if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+                           !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+                               WARNING(rt2x00dev, "Data pending for entry %u"
+                                       "in queue %u\n", entry->entry_idx, qid);
+                               return 1;
+                       }
+

                       if (rt2800usb_txdone_entry_check(entry, reg)) break;
               }

-               if (!entry || rt2x00queue_empty(queue))
-                       break;

drivers/net/wireless/rt2x00/rt2800usb.c +4 -2
-
-               rt2800_txdone_entry(entry, reg);
+               if (entry)
+                       rt2800_txdone_entry(entry, reg);
+next_reg:
+               kfifo_skip(&rt2x00dev->txstatus_fifo);

       }
+
+       return 0;
 }

 static void rt2800usb_work_txdone(struct work_struct *work)

drivers/net/wireless/rt2x00/rt2800usb.c +3 -1
@@ -545,7 +559,8 @@ static void rt2800usb_work_txdone(struct work_struct *work)
        struct data_queue *queue;
        struct queue_entry *entry;
 
-       rt2800usb_txdone(rt2x00dev);
+       if (rt2800usb_txdone(rt2x00dev))
+               goto out;
 
        /*
         * Process any trailing TX status reports for IO failures,
@@ -569,6 +584,7 @@ static void rt2800usb_work_txdone(struct work_struct *work)
                }
        }
 
+out:
        /*
         * The hw may delay sending the packet after DMA complete
         * if the medium is busy, thus the TX_STA_FIFO entry is
drivers/net/wireless/rt2x00/rt2x00usb.c +3 -4
index b6b4542..46c6d36 100644
@@ -265,14 +265,14 @@ static void rt2x00usb_interrupt_txdone(struct urb *urb)
        if (!test_and_clear_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags))
                return;
 
-       if (rt2x00dev->ops->lib->tx_dma_done)
-               rt2x00dev->ops->lib->tx_dma_done(entry);
-
        /*
         * Report the frame as DMA done
         */
        rt2x00lib_dmadone(entry);
 
+       if (rt2x00dev->ops->lib->tx_dma_done)
+               rt2x00dev->ops->lib->tx_dma_done(entry);
+
        /*
         * Check if the frame was correctly uploaded
         */
--

1.7.1

#2 lore raw

Hi Ivo

Quoted 9 lines · Ivo Van Doorn
Quoted 8 lines · Stanislaw Gruszka
drivers/net/wireless/rt2x00/rt2800usb.c +1 -1
-static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
+static int rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)

 {
       struct data_queue *queue;
       struct queue_entry *entry;
       u32 reg;
       u8 qid;

Quoted 8 lines
Quoted 2 lines

-       while (kfifo_get(&rt2x00dev->txstatus_fifo, &reg)) {
+       while (kfifo_peek(&rt2x00dev->txstatus_fifo, &reg)) {

I'm not too sure about this change, why do you need to do kfifo_peek and add gotos to the end of the while-loop to remove the item from the queue? There is no condition in which the obtained value from kfifo-peek will require it to be read again later (because when the value couldn't be handled we are throwing it away anyway using kfifo_skip).

There is new case (see below) where it is needed. I can get rid of goto, that will make code a bit cleaner. There is place for optimization, mainly make tx_status fifo per queue, but for now I just want to fix kernel crashes.

Quoted 21 lines
Quoted 21 lines

               /* TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus
                * qid is guaranteed to be one of the TX QIDs

drivers/net/wireless/rt2x00/rt2800usb.c +0 -0
@@ -517,25 +517,39 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
                if (unlikely(!queue)) {

                       WARNING(rt2x00dev, "Got TX status for an unavailable " "queue %u, dropping\n", qid);
-                       continue;
+                       goto next_reg;
               }

               /*
                * Inside each queue, we process each entry in a chronological
                * order. We first check that the queue is not empty. */
-               entry = NULL;
-               while (!rt2x00queue_empty(queue)) {
+               while (1) {
+                       entry = NULL;
+
+                       if (rt2x00queue_empty(queue))
+                               break;

Quoted 8 lines
Quoted 8 lines

+
                       entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE); +
+                       if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+                           !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+                               WARNING(rt2x00dev, "Data pending for entry %u"
+                                       "in queue %u\n", entry->entry_idx, qid);
+                               return 1;

Here is part of code where we exit the loop (and whole function) and do not remove head "reg" from tx_status fifo - and read it again when _txdone work is called next time.

Stanislaw

#3 lore raw

PATCH v2 1/1 rt2x00: rt2800usb: fix races in tx queue

We can perform parallel putting new entries in a queue (rt2x00queue_write_tx_frame()) and removing entries after finishing transmitting (rt2800usb_work_txdone()). There are cases when txdone may process an entry that was not fully send and nullify entry->skb. What in a result crash the kernel in rt2800usb_write_tx_desc() or rt2800usb_get_txwi() or other place where entry->skb is used.

To fix stop processing entries in txdone, which flags do not indicate transition was finished.

---
v1 -> v2: remove goto

 drivers/net/wireless/rt2x00/rt2800usb.c |   33 +++++++++++++++++++++++-------
 drivers/net/wireless/rt2x00/rt2x00usb.c |    6 ++--
 2 files changed, 28 insertions(+), 11 deletions(-)
drivers/net/wireless/rt2x00/rt2800usb.c +25 -8
index 5075593..670a30d 100644
@@ -500,14 +500,14 @@ static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg)
 	return true;
 }
 
-static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
+static int rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 {
 	struct data_queue *queue;
 	struct queue_entry *entry;
 	u32 reg;
 	u8 qid;
 
-	while (kfifo_get(&rt2x00dev->txstatus_fifo, &reg)) {
+	while (kfifo_peek(&rt2x00dev->txstatus_fifo, &reg)) {
 
 		/* TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus
 		 * qid is guaranteed to be one of the TX QIDs
@@ -517,6 +517,7 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 		if (unlikely(!queue)) {
 			WARNING(rt2x00dev, "Got TX status for an unavailable "
 					   "queue %u, dropping\n", qid);
+			kfifo_skip(&rt2x00dev->txstatus_fifo);
 			continue;
 		}
 
@@ -524,18 +525,32 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 		 * Inside each queue, we process each entry in a chronological
 		 * order. We first check that the queue is not empty.
 		 */
-		entry = NULL;
-		while (!rt2x00queue_empty(queue)) {
+		while (1) {
+			entry = NULL;
+
+			if (rt2x00queue_empty(queue))
+				break;
+
 			entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE);
+
+			if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+			    !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+				WARNING(rt2x00dev, "Data pending for entry %u"
+					"in queue %u\n", entry->entry_idx, qid);
+				return 1;
+			}
+
 			if (rt2800usb_txdone_entry_check(entry, reg))
 				break;
 		}
 
-		if (!entry || rt2x00queue_empty(queue))
-			break;
+		if (entry)
+			rt2800_txdone_entry(entry, reg);
 
-		rt2800_txdone_entry(entry, reg);
+		kfifo_skip(&rt2x00dev->txstatus_fifo);
 	}
+
+	return 0;
 }
 
 static void rt2800usb_work_txdone(struct work_struct *work)
@@ -545,7 +560,8 @@ static void rt2800usb_work_txdone(struct work_struct *work)
 	struct data_queue *queue;
 	struct queue_entry *entry;
 
-	rt2800usb_txdone(rt2x00dev);
+	if (rt2800usb_txdone(rt2x00dev))
+		goto out;
 
 	/*
 	 * Process any trailing TX status reports for IO failures,
@@ -569,6 +585,7 @@ static void rt2800usb_work_txdone(struct work_struct *work)
 		}
 	}
 
+out:
 	/*
 	 * The hw may delay sending the packet after DMA complete
 	 * if the medium is busy, thus the TX_STA_FIFO entry is
drivers/net/wireless/rt2x00/rt2x00usb.c +3 -3
index b6b4542..46c6d36 100644
@@ -265,14 +265,14 @@ static void rt2x00usb_interrupt_txdone(struct urb *urb)
 	if (!test_and_clear_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags))
 		return;
 
-	if (rt2x00dev->ops->lib->tx_dma_done)
-		rt2x00dev->ops->lib->tx_dma_done(entry);
-
 	/*
 	 * Report the frame as DMA done
 	 */
 	rt2x00lib_dmadone(entry);
 
+	if (rt2x00dev->ops->lib->tx_dma_done)
+		rt2x00dev->ops->lib->tx_dma_done(entry);
+
 	/*
 	 * Check if the frame was correctly uploaded
 	 */
Signature · 1 line
1.7.1
#4 lore raw

Hi Stanislaw,

Quoted 8 lines · Stanislaw Gruszka

We can perform parallel putting new entries in a queue (rt2x00queue_write_tx_frame()) and removing entries after finishing transmitting (rt2800usb_work_txdone()). There are cases when txdone may process an entry that was not fully send and nullify entry->skb. What in a result crash the kernel in rt2800usb_write_tx_desc() or rt2800usb_get_txwi() or other place where entry->skb is used.

Quoted 6 lines

To fix stop processing entries in txdone, which flags do not indicate transition was finished.

Sorry for responding this late. I just didn't find the bandwidth to look at this earlier :-(

First of all, I don't think stable should be cc-ed on this particular patch. Although the ideas of the patch are certainly applicable to the stable trees, this patch will simply not apply to 3.0 or earlier, since the code you are patching has been moved around between 3.0 and 3.1.

Second, I think it would be appropriate to split the patch in 2, or maybe 3, parts:
1. The hunk to rt2x00usb to reverse the entry flag handling and the tx dma done handling.
2. The hunk that checks that the entry on which the TX status is being reported has already been properly completed its TX done handling.
3. The remainder, i.e. the retrying of handling a TX status report if the entry hasn't been fully completed its TX done handling yet.

The code in this area has been proven to be very fragile, so I prefer to make mini changes to it in small steps, so that we can properly bisect which change exactly has caused a problem.

See further down for more thoughts.

Quoted 38 lines

---
v1 -> v2: remove goto

 drivers/net/wireless/rt2x00/rt2800usb.c |   33 +++++++++++++++++++++++-------
 drivers/net/wireless/rt2x00/rt2x00usb.c |    6 ++--
 2 files changed, 28 insertions(+), 11 deletions(-)
drivers/net/wireless/rt2x00/rt2800usb.c +3 -2
index 5075593..670a30d 100644
@@ -500,14 +500,14 @@ static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg)
 	return true;
 }
 
-static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
+static int rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 {
 	struct data_queue *queue;
 	struct queue_entry *entry;
 	u32 reg;
 	u8 qid;
 
-	while (kfifo_get(&rt2x00dev->txstatus_fifo, &reg)) {
+	while (kfifo_peek(&rt2x00dev->txstatus_fifo, &reg)) {
 
 		/* TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus
 		 * qid is guaranteed to be one of the TX QIDs
@@ -517,6 +517,7 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 		if (unlikely(!queue)) {
 			WARNING(rt2x00dev, "Got TX status for an unavailable "
 					   "queue %u, dropping\n", qid);
+			kfifo_skip(&rt2x00dev->txstatus_fifo);
 			continue;
 		}
 
@@ -524,18 +525,32 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 		 * Inside each queue, we process each entry in a chronological

drivers/net/wireless/rt2x00/rt2800usb.c static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev) · line 533

Quoted 10 lines
drivers/net/wireless/rt2x00/rt2800usb.c +6 -2
 		 * order. We first check that the queue is not empty.
 		 */
-		entry = NULL;
-		while (!rt2x00queue_empty(queue)) {
+		while (1) {
+			entry = NULL;
+
+			if (rt2x00queue_empty(queue))
+				break;
+

Do we really have to change this in a while (1) loop? I really hate those kind of loops, as it is quite easy to make an uneducated change later on and create a never ending loop. I don't think it is necessary to change the while loop conditions, if you just move the entry = NULL assignment to the end of the loop. Then you can keep the original while condition.

drivers/net/wireless/rt2x00/rt2800usb.c static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev) · line 542

Quoted 9 lines
drivers/net/wireless/rt2x00/rt2800usb.c +8 -0
 			entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE);
+
+			if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+			    !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+				WARNING(rt2x00dev, "Data pending for entry %u"
+					"in queue %u\n", entry->entry_idx, qid);
+				return 1;
+			}
+

Do we have to do this check inside the while loop?
Wouldn't it be enough to keep the loop as-is and have this check done after the while loop, on the found entry?

Quoted 47 lines
drivers/net/wireless/rt2x00/rt2800usb.c +8 -4
 			if (rt2800usb_txdone_entry_check(entry, reg))
 				break;
 		}
 
-		if (!entry || rt2x00queue_empty(queue))
-			break;
+		if (entry)
+			rt2800_txdone_entry(entry, reg);
 
-		rt2800_txdone_entry(entry, reg);
+		kfifo_skip(&rt2x00dev->txstatus_fifo);
 	}
+
+	return 0;
 }
 
 static void rt2800usb_work_txdone(struct work_struct *work)
@@ -545,7 +560,8 @@ static void rt2800usb_work_txdone(struct work_struct *work)
 	struct data_queue *queue;
 	struct queue_entry *entry;
 
-	rt2800usb_txdone(rt2x00dev);
+	if (rt2800usb_txdone(rt2x00dev))
+		goto out;
 
 	/*
 	 * Process any trailing TX status reports for IO failures,
@@ -569,6 +585,7 @@ static void rt2800usb_work_txdone(struct work_struct *work)
 		}
 	}
 
+out:
 	/*
 	 * The hw may delay sending the packet after DMA complete
 	 * if the medium is busy, thus the TX_STA_FIFO entry is
drivers/net/wireless/rt2x00/rt2x00usb.c +0 -3
index b6b4542..46c6d36 100644
@@ -265,14 +265,14 @@ static void rt2x00usb_interrupt_txdone(struct urb *urb)
 	if (!test_and_clear_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags))
 		return;
 
-	if (rt2x00dev->ops->lib->tx_dma_done)
-		rt2x00dev->ops->lib->tx_dma_done(entry);
-
 	/*

drivers/net/wireless/rt2x00/rt2x00usb.c static void rt2x00usb_interrupt_txdone(struct urb *urb) · line 278

Quoted 10 lines
drivers/net/wireless/rt2x00/rt2x00usb.c +3 -0
 	 * Report the frame as DMA done
 	 */
 	rt2x00lib_dmadone(entry);
 
+	if (rt2x00dev->ops->lib->tx_dma_done)
+		rt2x00dev->ops->lib->tx_dma_done(entry);
+
 	/*
 	 * Check if the frame was correctly uploaded
 	 */

---
Gertjan

#5 lore raw

Hello Gertjan

Quoted 5 lines · Gertjan van Wingerde

Sorry for responding this late. I just didn't find the bandwidth to look at this earlier :-(

First of all, I don't think stable should be cc-ed on this particular patch. Although the ideas of the patch are certainly applicable to the stable trees, this patch will simply not apply to 3.0 or earlier, since the code you are patching has been moved around between 3.0 and 3.1.

CCing stable mean that we want that fix in stable kernel, not that patch have to apply cleanly. When greg-kh will be applying patch on stable tries, I'll provide proper backport.

Quoted 11 lines

Second, I think it would be appropriate to split the patch in 2, or maybe 3, parts:
1. The hunk to rt2x00usb to reverse the entry flag handling and the tx dma done handling.
2. The hunk that checks that the entry on which the TX status is being reported has already been properly completed its TX done handling.
3. The remainder, i.e. the retrying of handling a TX status report if the entry hasn't been fully completed its TX done handling yet.

The code in this area has been proven to be very fragile, so I prefer to make mini changes to it in small steps, so that we can properly bisect which change exactly has caused a problem.

See further down for more thoughts.

Thanks for comments. I'll repost small patch that should fix the bug and don't do things you dislike.

Stanislaw

#6 lore raw
Quoted 20 lines · Stanislaw Gruszka

Hello Gertjan

Quoted 5 lines · Gertjan van Wingerde

Sorry for responding this late. I just didn't find the bandwidth to look at this earlier :-(

First of all, I don't think stable should be cc-ed on this particular patch. Although the ideas of the patch are certainly applicable to the stable trees, this patch will simply not apply to 3.0 or earlier, since the code you are patching has been moved around between 3.0 and 3.1.

CCing stable mean that we want that fix in stable kernel, not that patch have to apply cleanly. When greg-kh will be applying patch on stable tries, I'll provide proper backport.

Quoted 7 lines

Second, I think it would be appropriate to split the patch in 2, or maybe 3, parts:
1. The hunk to rt2x00usb to reverse the entry flag handling and the tx dma done handling.
2. The hunk that checks that the entry on which the TX status is being reported has already been properly completed its TX done handling.
3. The remainder, i.e. the retrying of handling a TX status report if the entry hasn't been fully completed its TX done handling yet.

Quoted 7 lines
Quoted 4 lines

The code in this area has been proven to be very fragile, so I prefer to make mini changes to it in small steps, so that we can properly bisect which change exactly has caused a problem.

See further down for more thoughts.

Thanks for comments. I'll repost small patch that should fix the bug and don't do things you dislike.

Hmm, I planed to post the below patch, but unfortunately it does not fix the crash on my system (rare reproducible after an hour of working). Seems there are more problems here. Looks like there is possibility to mishmash indexes i.e. make indexes like {Q_INDEX, Q_INDEX_DMA_DONE, Q_INDEX_DONE} = {44, 54, 44}, whereas they should be {44, 44, 44} or {45, 43, 41}. Original patch seems to preventing this (fix or mask the problem), but honestly I do not understand way. I have to look more closely at it.

Stanislaw

drivers/net/wireless/rt2x00/rt2800usb.c +9 -0
index 5075593..d8b27ae 100644
@@ -464,6 +464,15 @@ static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg)
 	int wcid, ack, pid;
 	int tx_wcid, tx_ack, tx_pid;
 
+	if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+	    !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+		WARNING(entry->queue->rt2x00dev,
+			"Data pending for entry %u in queue %u\n",
+			entry->entry_idx, entry->queue->qid);
+		cond_resched();
+		return false;
+	}
+
 	wcid	= rt2x00_get_field32(reg, TX_STA_FIFO_WCID);
 	ack	= rt2x00_get_field32(reg, TX_STA_FIFO_TX_ACK_REQUIRED);
 	pid	= rt2x00_get_field32(reg, TX_STA_FIFO_PID_TYPE);
drivers/net/wireless/rt2x00/rt2x00usb.c +2 -3
index b6b4542..32eb5aa 100644
@@ -265,14 +265,13 @@ static void rt2x00usb_interrupt_txdone(struct urb *urb)
 	if (!test_and_clear_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags))
 		return;
 
-	if (rt2x00dev->ops->lib->tx_dma_done)
-		rt2x00dev->ops->lib->tx_dma_done(entry);
-
 	/*
 	 * Report the frame as DMA done
 	 */
 	rt2x00lib_dmadone(entry);
 
+	if (rt2x00dev->ops->lib->tx_dma_done)
+		rt2x00dev->ops->lib->tx_dma_done(entry);
 	/*
 	 * Check if the frame was correctly uploaded
 	 */
#7 lore raw
Quoted 15 lines · Stanislaw Gruszka
Quoted 7 lines
Quoted 7 lines

Second, I think it would be appropriate to split the patch in 2, or maybe 3, parts:
1. The hunk to rt2x00usb to reverse the entry flag handling and the tx dma done handling.
2. The hunk that checks that the entry on which the TX status is being reported has already been properly completed its TX done handling.
3. The remainder, i.e. the retrying of handling a TX status report if the entry hasn't been fully completed its TX done handling yet.

Quoted 7 lines
Quoted 4 lines

The code in this area has been proven to be very fragile, so I prefer to make mini changes to it in small steps, so that we can properly bisect which change exactly has caused a problem.

See further down for more thoughts.

Thanks for comments. I'll repost small patch that should fix the bug and don't do things you dislike.

Quoted 7 lines

Hmm, I planed to post the below patch, but unfortunately it does not fix the crash on my system (rare reproducible after an hour of working). Seems there are more problems here. Looks like there is possibility to mishmash indexes i.e. make indexes like {Q_INDEX, Q_INDEX_DMA_DONE, Q_INDEX_DONE} = {44, 54, 44}, whereas they should be {44, 44, 44} or {45, 43, 41}. Original patch seems to preventing this (fix or mask the problem), but honestly I do not understand way. I have to look more closely at it.

Ok, I think I found these other problems, seems we have also check ENTRY_DATA_PENDING flags and add similar checks in rt2800usb_work_txdone when checking against failed I/O.

Justin, if you have opportunity test below patch (for 3.0 kernel). It does not crash here so far, but on my system bug is very rarely reproducible, so I have to test whole night or more to be sure.

Comments welcome. If patch is ok, I will split it into 2 parts and post officially.

drivers/net/wireless/rt2x00/rt2800lib.c +10 -0
index 2a6aa85..49a9c76 100644
@@ -607,6 +607,16 @@ static bool rt2800_txdone_entry_check(struct queue_entry *entry, u32 reg)
 	int wcid, ack, pid;
 	int tx_wcid, tx_ack, tx_pid;
 
+	if (test_bit(ENTRY_DATA_PENDING, &entry->flags) ||
+	    test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+	    !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+		WARNING(entry->queue->rt2x00dev,
+			"Data pending for entry %u in queue %u\n",
+			entry->entry_idx, entry->queue->qid);
+		cond_resched();
+		return false;
+	}
+
 	wcid	= rt2x00_get_field32(reg, TX_STA_FIFO_WCID);
 	ack	= rt2x00_get_field32(reg, TX_STA_FIFO_TX_ACK_REQUIRED);
 	pid	= rt2x00_get_field32(reg, TX_STA_FIFO_PID_TYPE);
drivers/net/wireless/rt2x00/rt2800usb.c +4 -1
index ba82c97..3dfb4f3 100644
@@ -477,8 +477,11 @@ static void rt2800usb_work_txdone(struct work_struct *work)
 		while (!rt2x00queue_empty(queue)) {
 			entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE);
 
-			if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags))
+			if (test_bit(ENTRY_DATA_PENDING, &entry->flags) ||
+			    test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
+			    !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags))
 				break;
+
 			if (test_bit(ENTRY_DATA_IO_FAILED, &entry->flags))
 				rt2x00lib_txdone_noinfo(entry, TXDONE_FAILURE);
 			else if (rt2x00queue_status_timeout(entry))
drivers/net/wireless/rt2x00/rt2x00usb.c +2 -3
index 8f90f62..7ec9e4f 100644
@@ -265,14 +265,13 @@ static void rt2x00usb_interrupt_txdone(struct urb *urb)
 	if (!test_and_clear_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags))
 		return;
 
-	if (rt2x00dev->ops->lib->tx_dma_done)
-		rt2x00dev->ops->lib->tx_dma_done(entry);
-
 	/*
 	 * Report the frame as DMA done
 	 */
 	rt2x00lib_dmadone(entry);
 
+	if (rt2x00dev->ops->lib->tx_dma_done)
+		rt2x00dev->ops->lib->tx_dma_done(entry);
 	/*
 	 * Check if the frame was correctly uploaded
 	 */
#8 lore raw
Quoted 28 lines · Stanislaw Gruszka
Quoted 15 lines · Stanislaw Gruszka
Quoted 7 lines
Quoted 7 lines

Second, I think it would be appropriate to split the patch in 2, or maybe 3, parts:
1. The hunk to rt2x00usb to reverse the entry flag handling and the tx dma done handling.
2. The hunk that checks that the entry on which the TX status is being reported has already been properly completed its TX done handling.
3. The remainder, i.e. the retrying of handling a TX status report if the entry hasn't been fully completed its TX done handling yet.

Quoted 7 lines
Quoted 4 lines

The code in this area has been proven to be very fragile, so I prefer to make mini changes to it in small steps, so that we can properly bisect which change exactly has caused a problem.

See further down for more thoughts.

Thanks for comments. I'll repost small patch that should fix the bug and don't do things you dislike.

Quoted 7 lines

Hmm, I planed to post the below patch, but unfortunately it does not fix the crash on my system (rare reproducible after an hour of working). Seems there are more problems here. Looks like there is possibility to mishmash indexes i.e. make indexes like {Q_INDEX, Q_INDEX_DMA_DONE, Q_INDEX_DONE} = {44, 54, 44}, whereas they should be {44, 44, 44} or {45, 43, 41}. Original patch seems to preventing this (fix or mask the problem), but honestly I do not understand way. I have to look more closely at it.

Ok, I think I found these other problems, seems we have also check ENTRY_DATA_PENDING flags and add similar checks in rt2800usb_work_txdone when checking against failed I/O.

Quoted 6 lines

Justin, if you have opportunity test below patch (for 3.0 kernel). It does not crash here so far, but on my system bug is very rarely reproducible, so I have to test whole night or more to be sure.

Comments welcome. If patch is ok, I will split it into 2 parts and post officially.

This patch crashes as well. I have to debug this issue a bit more ...

Stanislaw

#9 lore raw
Quoted 1 line · Stanislaw Gruszka

This patch crashes as well. I have to debug this issue a bit more ...

Figured this out, while(1) change helped in my original patch, without that we can have ops in rt2800usb_get_txwi. I'm going to post reworked patches now.

Stanislaw

#10 lore raw
In reply to Stanislaw Gruszka

Hi,

Quoted 48 lines · Ivo Van Doorn
Quoted 9 lines · Stanislaw Gruszka
Quoted 8 lines
drivers/net/wireless/rt2x00/rt2800usb.c +1 -1
-static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
+static int rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)

 {
       struct data_queue *queue;
       struct queue_entry *entry;
       u32 reg;
       u8 qid;

Quoted 8 lines
Quoted 2 lines

-       while (kfifo_get(&rt2x00dev->txstatus_fifo, &reg)) {
+       while (kfifo_peek(&rt2x00dev->txstatus_fifo, &reg)) {

I'm not too sure about this change, why do you need to do kfifo_peek and add gotos to the end of the while-loop to remove the item from the queue? There is no condition in which the obtained value from kfifo-peek will require it to be read again later (because when the value couldn't be handled we are throwing it away anyway using kfifo_skip).

There is new case (see below) where it is needed. I can get rid of goto, that will make code a bit cleaner. There is place for optimization, mainly make tx_status fifo per queue, but for now I just want to fix kernel crashes.

Quoted 25 lines
Quoted 25 lines

               /* TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus
                * qid is guaranteed to be one of the TX QIDs

drivers/net/wireless/rt2x00/rt2800usb.c +0 -0
@@ -517,25 +517,39 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
                if (unlikely(!queue)) {

                       WARNING(rt2x00dev, "Got TX status for an unavailable " "queue %u, dropping\n", qid);
-                       continue;
+                       goto next_reg;
               }

               /*
                * Inside each queue, we process each entry in a chronological
                * order. We first check that the queue is not empty. */
-               entry = NULL;
-               while (!rt2x00queue_empty(queue)) {
+               while (1) {
+                       entry = NULL;

drivers/net/wireless/rt2x00/rt2800usb.c +4 -0
+
+                       if (rt2x00queue_empty(queue))
+                               break;
+

                       entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE); +
+                       if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

+                           !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+                               WARNING(rt2x00dev, "Data pending for entry %u"
+                                       "in queue %u\n", entry->entry_idx, qid);
+                               return 1;

Here is part of code where we exit the loop (and whole function) and do not remove head "reg" from tx_status fifo - and read it again when _txdone work is called next time.

Well but for what reason would we want to read the register again? If we found an status report
for a queue which does not have pending items, then in this change it would mean that the
status report is intended for a TX frame which has yet to be enqueued to the hardware.

Obviously this means a mismatch between the TX status report and the actual frame to which it
is being connected.

Ivo

#11 lore raw
Quoted 10 lines · Ivo Van Doorn

Well but for what reason would we want to read the register again? If we found an status report
for a queue which does not have pending items, then in this change it would mean that the
status report is intended for a TX frame which has yet to be enqueued to the hardware.

Obviously this means a mismatch between the TX status report and the actual frame to which it
is being connected.

Ok, I will rewrite patch to skip TX status fifo reg if entry is pending.

Stanislaw

#12 lore raw
In reply to Ivo Van Doorn
Quoted 66 lines · Ivo Van Doorn

Hi,

Quoted 48 lines · Ivo Van Doorn
Quoted 9 lines · Stanislaw Gruszka
Quoted 8 lines
drivers/net/wireless/rt2x00/rt2800usb.c +1 -1
-static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
+static int rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
 {
       struct data_queue *queue;
       struct queue_entry *entry;
       u32 reg;
       u8 qid;
Quoted 8 lines

drivers/net/wireless/rt2x00/rt2800usb.c static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg) · line 510

Quoted 2 lines
drivers/net/wireless/rt2x00/rt2800usb.c +1 -1
-       while (kfifo_get(&rt2x00dev->txstatus_fifo, &reg)) {
+       while (kfifo_peek(&rt2x00dev->txstatus_fifo, &reg)) {

I'm not too sure about this change, why do you need to do kfifo_peek and add gotos to the end of the while-loop to remove the item from the queue? There is no condition in which the obtained value from kfifo-peek will require it to be read again later (because when the value couldn't be handled we are throwing it away anyway using kfifo_skip).

There is new case (see below) where it is needed. I can get rid of goto, that will make code a bit cleaner. There is place for optimization, mainly make tx_status fifo per queue, but for now I just want to fix kernel crashes.

Quoted 25 lines
Quoted 25 lines

/* TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus
* qid is guaranteed to be one of the TX QIDs

drivers/net/wireless/rt2x00/rt2800usb.c +9 -3
@@ -517,25 +517,39 @@ static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
               if (unlikely(!queue)) {
                       WARNING(rt2x00dev, "Got TX status for an unavailable "
                                          "queue %u, dropping\n", qid);
-                       continue;
+                       goto next_reg;
               }
 
               /*
                * Inside each queue, we process each entry in a chronological
                * order. We first check that the queue is not empty.
                */
-               entry = NULL;
-               while (!rt2x00queue_empty(queue)) {
+               while (1) {
+                       entry = NULL;
+
+                       if (rt2x00queue_empty(queue))
+                               break;
+
                       entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE);
+
+                       if (test_bit(ENTRY_OWNER_DEVICE_DATA, &entry->flags) ||
Quoted 8 lines
Quoted 4 lines

drivers/net/wireless/rt2x00/rt2800usb.c static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev) · line 539

Quoted 4 lines
drivers/net/wireless/rt2x00/rt2800usb.c +4 -0
+                           !test_bit(ENTRY_DATA_STATUS_PENDING, &entry->flags)) {
+                               WARNING(rt2x00dev, "Data pending for entry %u"
+                                       "in queue %u\n", entry->entry_idx, qid);
+                               return 1;

Here is part of code where we exit the loop (and whole function) and do not remove head "reg" from tx_status fifo - and read it again when _txdone work is called next time.

Well but for what reason would we want to read the register again? If we found an status report
for a queue which does not have pending items, then in this change it would mean that the
status report is intended for a TX frame which has yet to be enqueued to the hardware.

Quoted 3 lines

Obviously this means a mismatch between the TX status report and the actual frame to which it
is being connected.

Hmm, if I understood the patch description correctly, then there may be a race in the code, where we actually read the TX status report before we have had the chance to handle the TX done URB interrupt. As far as I understood it, it are not spurious TX status reports for frames that were never submitted to the HW.
If that is really the case then it is quite reasonable to wait a little bit until the TX done code has been executed and then process the TX status report again.
However, we must be damn sure that this is really what happens.

Stanislaw, please confirm (or deny) that my understanding is correct.

Note that I have some comments of my own to the patch, which I will send as a response to the v2 patch.

---
Gertjan

#13 lore raw
Quoted 23 lines · Gertjan van Wingerde
Quoted 11 lines
Quoted 3 lines

Here is part of code where we exit the loop (and whole function) and do not remove head "reg" from tx_status fifo - and read it again when _txdone work is called next time.

Well but for what reason would we want to read the register again? If we found an status report
for a queue which does not have pending items, then in this change it would mean that the
status report is intended for a TX frame which has yet to be enqueued to the hardware.

Quoted 3 lines

Obviously this means a mismatch between the TX status report and the actual frame to which it
is being connected.

Hmm, if I understood the patch description correctly, then there may be a race in the code, where we actually read the TX status report before we have had the chance to handle the TX done URB interrupt. As far as I understood it, it are not spurious TX status reports for frames that were never submitted to the HW.
If that is really the case then it is quite reasonable to wait a little bit until the TX done code has been executed and then process the TX status report again.
However, we must be damn sure that this is really what happens.

Quoted 1 line

Stanislaw, please confirm (or deny) that my understanding is correct.

In this loop

                while (!rt2x00queue_empty(queue)) {
                        entry = rt2x00queue_get_entry(queue,Q_INDEX_DONE);
                        if (rt2800usb_txdone_entry_check(entry, reg))
                                break;
                }

we can process entry that is currently added to tx queue and nulify skb.

This can happen if we return false from rt2800usb_txdone_entry_check(), so only when reg does not match ((wcid != tx_wcid) || (ack != tx_ack) || (pid != tx_pid)), or previous entry is ENTRY_DATA_IO_FAILED.

Stanislaw