Skip to contentSkip to conversation

net: dwc_eth_qos: Fully rewrite RX descriptor field 3

PATCH New Marek Vasut 4 participants 31 messages Mar 23, 2020 – Apr 29, 2020 Network drivers

lore
Share & tools
lore
#0 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

The RX descriptor field 3 should contain only OWN and BUF1V bits before being used for receiving data by the DMA engine. However, right now, if the descriptor was already used for receiving data and is being cleared, the field 3 is only modified and the aforementioned two bits are ORRed into the field. This could lead to a residual dirty bits being left in the field 3 from previous transfer, and it generally does. Fully set the field 3 instead to clear those residual dirty bits.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/dwc_eth_qos.c +2 -2
index 0564bebf76..4bce6d4290 100644
@@ -1242,7 +1242,7 @@ static int eqos_start(struct udevice *dev)
 		struct eqos_desc *rx_desc = &(eqos->rx_descs[i]);
 		rx_desc->des0 = (u32)(ulong)(eqos->rx_dma_buf +
 					     (i * EQOS_MAX_PACKET_SIZE));
-		rx_desc->des3 |= EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
+		rx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
 	}
 	eqos->config->ops->eqos_flush_desc(eqos->descs);
 
@@ -1436,7 +1436,7 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 	 * writes to the rest of the descriptor too.
 	 */
 	mb();
-	rx_desc->des3 |= EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
+	rx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
 	eqos->config->ops->eqos_flush_desc(rx_desc);
 
 	writel((ulong)rx_desc, &eqos->dma_regs->ch0_rxdesc_tail_pointer);
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

Quoted 8 lines · Marek Vasut

The RX descriptor field 3 should contain only OWN and BUF1V bits before being used for receiving data by the DMA engine. However, right now, if the descriptor was already used for receiving data and is being cleared, the field 3 is only modified and the aforementioned two bits are ORRed into the field. This could lead to a residual dirty bits being left in the field 3 from previous transfer, and it generally does. Fully set the field 3 instead to clear those residual dirty bits.

Quoted 6 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

Joe, this was posted for a month, can this go in ?

#2 lore raw

PATCH 2/6 net: dwc_eth_qos: Correctly wrap around TX descriptor tail pointer

In reply to Marek Vasut

This code programs the next descriptor in the TX descriptor ring into the hardware as the last valid TX descriptor. The problem is that if the currenty descriptor is the last one in the array, the code will not wrap around correctly and use TX descriptor 0 again, but instead will use TX descriptor at address right past the TX descriptor ring, which is the first descriptor in the RX ring.

Fix this by adding the necessary wrap-around.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +2 -1
index 4bce6d4290..c4f665bda9 100644
@@ -1373,7 +1373,8 @@ static int eqos_send(struct udevice *dev, void *packet, int length)
 	tx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_FD | EQOS_DESC3_LD | length;
 	eqos->config->ops->eqos_flush_desc(tx_desc);
 
-	writel((ulong)(tx_desc + 1), &eqos->dma_regs->ch0_txdesc_tail_pointer);
+	writel((ulong)(&(eqos->tx_descs[eqos->tx_desc_idx])),
+		&eqos->dma_regs->ch0_txdesc_tail_pointer);
 
 	for (i = 0; i < 1000000; i++) {
 		eqos->config->ops->eqos_inval_desc(tx_desc);
Signature · 1 line
2.25.1
#3 lore raw

PATCH 2/6 net: dwc_eth_qos: Correctly wrap around TX descriptor tail pointer

Quoted 35 lines · Marek Vasut

This code programs the next descriptor in the TX descriptor ring into the hardware as the last valid TX descriptor. The problem is that if the currenty descriptor is the last one in the array, the code will not wrap around correctly and use TX descriptor 0 again, but instead will use TX descriptor at address right past the TX descriptor ring, which is the first descriptor in the RX ring.

Fix this by adding the necessary wrap-around.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +2 -2
index 4bce6d4290..c4f665bda9 100644
@@ -1373,7 +1373,8 @@ static int eqos_send(struct udevice *dev, void *packet, int length)
        tx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_FD | EQOS_DESC3_LD | length;
        eqos->config->ops->eqos_flush_desc(tx_desc);
 
-       writel((ulong)(tx_desc + 1), &eqos->dma_regs->ch0_txdesc_tail_pointer);
+       writel((ulong)(&(eqos->tx_descs[eqos->tx_desc_idx])),
+               &eqos->dma_regs->ch0_txdesc_tail_pointer);
 
        for (i = 0; i < 1000000; i++) {
                eqos->config->ops->eqos_inval_desc(tx_desc);
--
Quoted 2 lines

2.25.1

#4 lore raw

PATCH 2/6 net: dwc_eth_qos: Correctly wrap around TX descriptor tail pointer

In reply to Marek Vasut

Dear Marek,

Quoted 11 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45

This code programs the next descriptor in the TX descriptor ring into the hardware as the last valid TX descriptor. The problem is that if the currenty descriptor is the last one in the array, the code will not wrap around correctly and use TX descriptor 0 again, but instead will use TX descriptor at address right past the TX descriptor ring, which is the first descriptor in the RX ring.

Fix this by adding the necessary wrap-around.

Quoted 7 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

---

Thanks

Patrick

#5 lore raw

PATCH 3/6 net: dwc_eth_qos: Flush the RX descriptors on init

In reply to Marek Vasut

Currently the code only flushes the first RX descriptor, not every entry in the RX descriptor ring. Fix this, to make sure the DMA engine can pick the RX descriptors correctly.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +1 -1
index c4f665bda9..66cc301c8c 100644
@@ -1243,8 +1243,8 @@ static int eqos_start(struct udevice *dev)
 		rx_desc->des0 = (u32)(ulong)(eqos->rx_dma_buf +
 					     (i * EQOS_MAX_PACKET_SIZE));
 		rx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
+		eqos->config->ops->eqos_flush_desc(rx_desc);
 	}
-	eqos->config->ops->eqos_flush_desc(eqos->descs);
 
 	writel(0, &eqos->dma_regs->ch0_txdesc_list_haddress);
 	writel((ulong)eqos->tx_descs, &eqos->dma_regs->ch0_txdesc_list_address);
Signature · 1 line
2.25.1
#6 lore raw

PATCH 3/6 net: dwc_eth_qos: Flush the RX descriptors on init

Quoted 30 lines · Marek Vasut

Currently the code only flushes the first RX descriptor, not every entry in the RX descriptor ring. Fix this, to make sure the DMA engine can pick the RX descriptors correctly.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +1 -2
index c4f665bda9..66cc301c8c 100644
@@ -1243,8 +1243,8 @@ static int eqos_start(struct udevice *dev)
                rx_desc->des0 = (u32)(ulong)(eqos->rx_dma_buf +
                                             (i * EQOS_MAX_PACKET_SIZE));
                rx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
+               eqos->config->ops->eqos_flush_desc(rx_desc);
        }
-       eqos->config->ops->eqos_flush_desc(eqos->descs);
 
        writel(0, &eqos->dma_regs->ch0_txdesc_list_haddress);
        writel((ulong)eqos->tx_descs, &eqos->dma_regs->ch0_txdesc_list_address);
--
Quoted 2 lines

2.25.1

#7 lore raw

PATCH 3/6 net: dwc_eth_qos: Flush the RX descriptors on init

In reply to Marek Vasut

Dear Marek,

Quoted 14 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45
To: u-boot at lists.denx.de
Cc: Marek Vasut <[email protected]>; Joe Hershberger <[email protected]>; Patrice CHOTARD <[email protected]>; Patrick DELAUNAY <[email protected]>; Ramon Fried <[email protected]>; Stephen Warren <[email protected]>
Subject: [PATCH 3/6] net: dwc_eth_qos: Flush the RX descriptors on init
Importance: High

Currently the code only flushes the first RX descriptor, not every entry in the RX descriptor ring. Fix this, to make sure the DMA engine can pick the RX descriptors correctly.

Quoted 7 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

---

Thanks

Patrick

#8 lore raw

PATCH 4/6 net: dwc_eth_qos: Invalidate RX descriptor before reading

In reply to Marek Vasut

The current code polls the RX desciptor ring for new packets by reading the RX descriptor status. This works by accident, as the RX descriptors are often in non-cacheable memory. However, the driver does support use of RX descriptors in cacheable memory.

This patch adds a missing RX descriptor invalidation, which assures the CPU will read a fresh copy of the RX descriptor instead of a cached one.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/dwc_eth_qos.c +1 -0
index 66cc301c8c..e40c461278 100644
@@ -1397,6 +1397,7 @@ static int eqos_recv(struct udevice *dev, int flags, uchar **packetp)
 	debug("%s(dev=%p, flags=%x):\n", __func__, dev, flags);
 
 	rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
+	eqos->config->ops->eqos_inval_desc(rx_desc);
 	if (rx_desc->des3 & EQOS_DESC3_OWN) {
 		debug("%s: RX packet not available\n", __func__);
 		return -EAGAIN;
Signature · 1 line
2.25.1
#9 lore raw

PATCH 4/6 net: dwc_eth_qos: Invalidate RX descriptor before reading

Quoted 32 lines · Marek Vasut

The current code polls the RX desciptor ring for new packets by reading the RX descriptor status. This works by accident, as the RX descriptors are often in non-cacheable memory. However, the driver does support use of RX descriptors in cacheable memory.

This patch adds a missing RX descriptor invalidation, which assures the CPU will read a fresh copy of the RX descriptor instead of a cached one.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/dwc_eth_qos.c +1 -1
index 66cc301c8c..e40c461278 100644
@@ -1397,6 +1397,7 @@ static int eqos_recv(struct udevice *dev, int flags, uchar **packetp)
        debug("%s(dev=%p, flags=%x):\n", __func__, dev, flags);
 
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
+       eqos->config->ops->eqos_inval_desc(rx_desc);
        if (rx_desc->des3 & EQOS_DESC3_OWN) {
                debug("%s: RX packet not available\n", __func__);
                return -EAGAIN;
--
Quoted 2 lines

2.25.1

#10 lore raw

PATCH 4/6 net: dwc_eth_qos: Invalidate RX descriptor before reading

In reply to Marek Vasut

Dear Marek,

Quoted 11 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45

The current code polls the RX desciptor ring for new packets by reading the RX descriptor status. This works by accident, as the RX descriptors are often in non- cacheable memory. However, the driver does support use of RX descriptors in cacheable memory.

This patch adds a missing RX descriptor invalidation, which assures the CPU will read a fresh copy of the RX descriptor instead of a cached one.

Quoted 7 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

---

Patrick

#11 lore raw

PATCH 5/6 net: dwc_eth_qos: Invalidate RX packet DMA buffer

In reply to Marek Vasut

This patch prevents an issue where the RX packet might have been accessed by the CPU, which now has cached data from the packet in the caches and possibly various write buffers, and these data may be evicted from the caches into the DRAM while the buffer is also written by the DMA.

By invalidating the buffer after the CPU accessed it and before the DMA populates the buffer, it is assured that the buffer will not be corrupted.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/dwc_eth_qos.c +6 -0
index e40c461278..7dadb10fe7 100644
@@ -1430,6 +1430,9 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 	}
 
 	rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
+
+	eqos->config->ops->eqos_inval_buffer(packet, length);
+
 	rx_desc->des0 = (u32)(ulong)packet;
 	rx_desc->des1 = 0;
 	rx_desc->des2 = 0;
@@ -1492,6 +1495,9 @@ static int eqos_probe_resources_core(struct udevice *dev)
 	}
 	debug("%s: rx_pkt=%p\n", __func__, eqos->rx_pkt);
 
+	eqos->config->ops->eqos_inval_buffer(eqos->rx_dma_buf,
+			EQOS_MAX_PACKET_SIZE * EQOS_DESCRIPTORS_RX);
+
 	debug("%s: OK\n", __func__);
 	return 0;
 
Signature · 1 line
2.25.1
#12 lore raw

PATCH 5/6 net: dwc_eth_qos: Invalidate RX packet DMA buffer

Quoted 46 lines · Marek Vasut

This patch prevents an issue where the RX packet might have been accessed by the CPU, which now has cached data from the packet in the caches and possibly various write buffers, and these data may be evicted from the caches into the DRAM while the buffer is also written by the DMA.

By invalidating the buffer after the CPU accessed it and before the DMA populates the buffer, it is assured that the buffer will not be corrupted.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/dwc_eth_qos.c +6 -1
index e40c461278..7dadb10fe7 100644
@@ -1430,6 +1430,9 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
        }
 
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
+
+       eqos->config->ops->eqos_inval_buffer(packet, length);
+
        rx_desc->des0 = (u32)(ulong)packet;
        rx_desc->des1 = 0;
        rx_desc->des2 = 0;
@@ -1492,6 +1495,9 @@ static int eqos_probe_resources_core(struct udevice *dev)
        }
        debug("%s: rx_pkt=%p\n", __func__, eqos->rx_pkt);
 
+       eqos->config->ops->eqos_inval_buffer(eqos->rx_dma_buf,
+                       EQOS_MAX_PACKET_SIZE * EQOS_DESCRIPTORS_RX);
+
        debug("%s: OK\n", __func__);
        return 0;
 
--
Quoted 2 lines

2.25.1

#13 lore raw

PATCH 5/6 net: dwc_eth_qos: Invalidate RX packet DMA buffer

In reply to Marek Vasut

Hi Marek,

Quoted 11 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45

This patch prevents an issue where the RX packet might have been accessed by the CPU, which now has cached data from the packet in the caches and possibly various write buffers, and these data may be evicted from the caches into the DRAM while the buffer is also written by the DMA.

By invalidating the buffer after the CPU accessed it and before the DMA populates the buffer, it is assured that the buffer will not be corrupted.

Quoted 7 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

---

Patrick

#14 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

In reply to Marek Vasut
Quoted 42 lines · Marek Vasut

The RX descriptor field 3 should contain only OWN and BUF1V bits before being used for receiving data by the DMA engine. However, right now, if the descriptor was already used for receiving data and is being cleared, the field 3 is only modified and the aforementioned two bits are ORRed into the field. This could lead to a residual dirty bits being left in the field 3 from previous transfer, and it generally does. Fully set the field 3 instead to clear those residual dirty bits.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/dwc_eth_qos.c +2 -3
index 0564bebf76..4bce6d4290 100644
@@ -1242,7 +1242,7 @@ static int eqos_start(struct udevice *dev)
                struct eqos_desc *rx_desc = &(eqos->rx_descs[i]);
                rx_desc->des0 = (u32)(ulong)(eqos->rx_dma_buf +
                                             (i * EQOS_MAX_PACKET_SIZE));
-               rx_desc->des3 |= EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
+               rx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
        }
        eqos->config->ops->eqos_flush_desc(eqos->descs);
 
@@ -1436,7 +1436,7 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
         * writes to the rest of the descriptor too.
         */
        mb();
-       rx_desc->des3 |= EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
+       rx_desc->des3 = EQOS_DESC3_OWN | EQOS_DESC3_BUF1V;
        eqos->config->ops->eqos_flush_desc(rx_desc);
 
        writel((ulong)rx_desc, &eqos->dma_regs->ch0_rxdesc_tail_pointer);
--
Quoted 2 lines

2.25.1

#15 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

In reply to Marek Vasut
Quoted 7 lines · Marek Vasut

The RX descriptor field 3 should contain only OWN and BUF1V bits before being used for receiving data by the DMA engine. However, right now, if the descriptor was already used for receiving data and is being cleared, the field 3 is only modified and the aforementioned two bits are ORRed into the field. This could lead to a residual dirty bits being left in the field 3 from previous transfer, and it generally does. Fully set the field 3 instead to clear those residual dirty bits.

The series,

(On Jetson, using test/py tests)

#16 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

In reply to Marek Vasut

Dear Marek,

Quoted 21 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45

The RX descriptor field 3 should contain only OWN and BUF1V bits before being used for receiving data by the DMA engine. However, right now, if the descriptor was already used for receiving data and is being cleared, the field 3 is only modified and the aforementioned two bits are ORRed into the field. This could lead to a residual dirty bits being left in the field 3 from previous transfer, and it generally does. Fully set the field 3 instead to clear those residual dirty bits.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/dwc_eth_qos.c index renamed from drivers/net/dwc_eth_qos.c +0 -0
Quoted 2 lines

0564bebf76..4bce6d4290 100644
--- a/drivers/net/dwc_eth_qos.c

Thanks

Patrick

#17 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

In reply to Marek Vasut

Dear Marek,

Quoted 10 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45

The RX descriptor field 3 should contain only OWN and BUF1V bits before being used for receiving data by the DMA engine. However, right now, if the descriptor was already used for receiving data and is being cleared, the field 3 is only modified and the aforementioned two bits are ORRed into the field. This could lead to a residual dirty bits being left in the field 3 from previous transfer, and it generally does. Fully set the field 3 instead to clear those residual dirty bits.

Quoted 7 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

---

For the series,

Regards

Patrick

#18 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

Dear Marek,

Quoted 24 lines

From: Patrick DELAUNAY
Sent: lundi 6 avril 2020 17:11

Dear Marek,

Quoted 9 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45

The RX descriptor field 3 should contain only OWN and BUF1V bits before being used for receiving data by the DMA engine. However, right now, if the descriptor was already used for receiving data and is being cleared, the field 3 is only modified and the aforementioned two bits are ORRed into the field. This could lead to a residual dirty bits being left in the field 3 from previous transfer, and it generally does. Fully

set the field 3 instead to clear those residual dirty bits.

Quoted 8 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

---

Quoted 3 lines

For the series,

manual test: dhcp and run bootcmd_pxe)

To complete my test and to check the cache management in the driver,

I test the sequence (CONFIG_SYS_NONCACHED_MEMORY is activated):

1) ping with dcache ON: Always OK

STM32MP> dhcp
STM32MP> ping 91.162.57.93

2) ping with dcache OFF : Always OK

STM32MP> dcache off
STM32MP> ping 91.162.57.93

3) ping with dcache ON : Failed

STM32MP> dcache on
STM32MP> ping 91.162.57.93

Failed with "eqos_send: TX timeout"
and after "eqos_recv: RX packet not available"

4) ping with dcache OFF : Always OK

STM32MP> dcache on
STM32MP> ping 91.162.57.93

I don't sure this sequence is really valid or if this issue show a problem in cache management.

As I don't see any obvious issue in eqos_send, do you idea on reason of this issue ?

Quoted 3 lines

Regards

Patrick

#19 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

Quoted 1 line · Patrick DELAUNAY

Dear Marek,

Hi,

Quoted 28 lines

To complete my test and to check the cache management in the driver,

I test the sequence (CONFIG_SYS_NONCACHED_MEMORY is activated):

1) ping with dcache ON: Always OK

STM32MP> dhcp
STM32MP> ping 91.162.57.93

2) ping with dcache OFF : Always OK

STM32MP> dcache off
STM32MP> ping 91.162.57.93

3) ping with dcache ON : Failed

STM32MP> dcache on
STM32MP> ping 91.162.57.93

Failed with "eqos_send: TX timeout"
and after "eqos_recv: RX packet not available"

4) ping with dcache OFF : Always OK

STM32MP> dcache on
STM32MP> ping 91.162.57.93

Quoted 5 lines

I don't sure this sequence is really valid or if this issue show a problem in cache management.

As I don't see any obvious issue in eqos_send, do you idea on reason of this issue ?

So the test is basically:

dhcp
while true ; do
ping 192.168.1.300 ;
dcache off ;
ping 192.168.1.300 ;
dcache on ;
done

right ?

I can tell you what you're likely gonna see if you debug it further.

Use the board on an isolated network segment (get some USB ethernet or whatever), disable dhcp server, avahi and all other such stuff. Then run wireshark on that interface.

Then, patch the dwmac driver to dump every packet it sends and receives, the first 16 or so bytes are enough, so you'd see the header. Then run your test.

After a while, you will see that the DWMAC transmits some valid packet, but receives a packet with the exact same source MAC it just transmitted , the source MAC of the stm32 board. But if I recall it correctly, you won't see that packet on the line in wireshark, it just somehow loops back in.

You can even accelerate that process by using arping from the host PC, just arping the stm32 and the error will happen faster.

I was hoping I managed to fix all of these problems in this series, but apparently there is more :-( Do you feel like debugging this further?

#20 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

Dear,

Quoted 52 lines

From: Marek Vasut <[email protected]>
Sent: mardi 7 avril 2020 12:04

Quoted 1 line · Patrick DELAUNAY

Dear Marek,

Hi,

Quoted 28 lines

To complete my test and to check the cache management in the driver,

I test the sequence (CONFIG_SYS_NONCACHED_MEMORY is activated):

1) ping with dcache ON: Always OK

STM32MP> dhcp
STM32MP> ping 91.162.57.93

2) ping with dcache OFF : Always OK

STM32MP> dcache off
STM32MP> ping 91.162.57.93

3) ping with dcache ON : Failed

STM32MP> dcache on
STM32MP> ping 91.162.57.93

Failed with "eqos_send: TX timeout"
and after "eqos_recv: RX packet not available"

4) ping with dcache OFF : Always OK

STM32MP> dcache on
STM32MP> ping 91.162.57.93

Quoted 5 lines

I don't sure this sequence is really valid or if this issue show a problem in cache management.

As I don't see any obvious issue in eqos_send, do you idea on reason of this issue ?

So the test is basically:

dhcp
while true ; do
ping 192.168.1.300 ;
dcache off ;
ping 192.168.1.300 ;
dcache on ;
done

Quoted 1 line

right ?

Yes, exactly and it is more clear

Quoted 15 lines

I can tell you what you're likely gonna see if you debug it further.

Use the board on an isolated network segment (get some USB ethernet or whatever), disable dhcp server, avahi and all other such stuff. Then run wireshark on that interface.

Then, patch the dwmac driver to dump every packet it sends and receives, the first 16 or so bytes are enough, so you'd see the header. Then run your test.

After a while, you will see that the DWMAC transmits some valid packet, but receives a packet with the exact same source MAC it just transmitted , the source MAC of the stm32 board. But if I recall it correctly, you won't see that packet on the line in wireshark, it just somehow loops back in.

Quoted 5 lines

You can even accelerate that process by using arping from the host PC, just arping the stm32 and the error will happen faster.

I was hoping I managed to fix all of these problems in this series, but apparently there is more :-( Do you feel like debugging this further?

Yes it is my fear.

Thanks for advices...
I will try to debug it
but my bandwith is limited for the next week, so I can't give a guarantee.

Regards

Patrick

#21 lore raw

PATCH 1/6 net: dwc_eth_qos: Fully rewrite RX descriptor field 3

Quoted 76 lines · Patrick DELAUNAY

Dear,

Quoted 52 lines

From: Marek Vasut <[email protected]>
Sent: mardi 7 avril 2020 12:04

Quoted 1 line · Patrick DELAUNAY

Dear Marek,

Hi,

Quoted 28 lines

To complete my test and to check the cache management in the driver,

I test the sequence (CONFIG_SYS_NONCACHED_MEMORY is activated):

1) ping with dcache ON: Always OK

STM32MP> dhcp
STM32MP> ping 91.162.57.93

2) ping with dcache OFF : Always OK

STM32MP> dcache off
STM32MP> ping 91.162.57.93

3) ping with dcache ON : Failed

STM32MP> dcache on
STM32MP> ping 91.162.57.93

Failed with "eqos_send: TX timeout"
and after "eqos_recv: RX packet not available"

4) ping with dcache OFF : Always OK

STM32MP> dcache on
STM32MP> ping 91.162.57.93

Quoted 5 lines

I don't sure this sequence is really valid or if this issue show a problem in cache management.

As I don't see any obvious issue in eqos_send, do you idea on reason of this issue ?

So the test is basically:

dhcp
while true ; do
ping 192.168.1.300 ;
dcache off ;
ping 192.168.1.300 ;
dcache on ;
done

Quoted 1 line

right ?

Yes, exactly and it is more clear

Quoted 18 lines

I can tell you what you're likely gonna see if you debug it further.

Use the board on an isolated network segment (get some USB ethernet or whatever), disable dhcp server, avahi and all other such stuff. Then run wireshark on that interface.

Then, patch the dwmac driver to dump every packet it sends and receives, the first 16 or so bytes are enough, so you'd see the header. Then run your test.

After a while, you will see that the DWMAC transmits some valid packet, but receives a packet with the exact same source MAC it just transmitted , the source MAC of the stm32 board. But if I recall it correctly, you won't see that packet on the line in wireshark, it just somehow loops back in.

You can even accelerate that process by using arping from the host PC, just arping the stm32 and the error will happen faster.

Quoted 8 lines
Quoted 2 lines

I was hoping I managed to fix all of these problems in this series, but apparently there is more :-( Do you feel like debugging this further?

Yes it is my fear.

Thanks for advices...
I will try to debug it
but my bandwith is limited for the next week, so I can't give a guarantee.

I sent a patch to pad the descriptors to cacheline size just now, however I still didn't get Joe to review/apply any of the networking patches. I hope he's just busy with other stuff ...

#22 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

In reply to Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +3 -1
index 7dadb10fe7..c86b9d59a5 100644
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
 	rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
+	rx_desc->des0 = 0;
+	mb();
+	eqos->config->ops->eqos_flush_desc(rx_desc);
 	eqos->config->ops->eqos_inval_buffer(packet, length);
-
 	rx_desc->des0 = (u32)(ulong)packet;
 	rx_desc->des1 = 0;
 	rx_desc->des2 = 0;
Signature · 1 line
2.25.1
#23 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

Quoted 21 lines · Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +0 -0
index 7dadb10fe7..c86b9d59a5 100644

drivers/net/dwc_eth_qos.c static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length) · line 1435

Quoted 5 lines
drivers/net/dwc_eth_qos.c +2 -0
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
+       rx_desc->des0 = 0;
+       mb();

Better use wmb() for better understanding your goal here ?

Quoted 9 lines
drivers/net/dwc_eth_qos.c +1 -2
+       eqos->config->ops->eqos_flush_desc(rx_desc);
        eqos->config->ops->eqos_inval_buffer(packet, length);
-
        rx_desc->des0 = (u32)(ulong)packet;
        rx_desc->des1 = 0;
        rx_desc->des2 = 0;
--

2.25.1

#24 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

Quoted 23 lines · Ramon Fried
Quoted 22 lines · Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +0 -0
index 7dadb10fe7..c86b9d59a5 100644
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
Quoted 5 lines

drivers/net/dwc_eth_qos.c static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length) · line 1435

Quoted 4 lines
drivers/net/dwc_eth_qos.c +2 -0
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
+       rx_desc->des0 = 0;
+       mb();

Better use wmb() for better understanding your goal here ?

The driver uses mb() all over the place, so I'm just keeping it consistent here.

But I wonder whether we even need all those barriers in this driver at all.

#25 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

Quoted 27 lines · Marek Vasut
Quoted 25 lines · Ramon Fried
Quoted 24 lines · Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +0 -0
index 7dadb10fe7..c86b9d59a5 100644
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
Quoted 8 lines
Quoted 3 lines

drivers/net/dwc_eth_qos.c static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length) · line 1435

Quoted 2 lines
drivers/net/dwc_eth_qos.c +2 -0
+       rx_desc->des0 = 0;
+       mb();

Better use wmb() for better understanding your goal here ?

The driver uses mb() all over the place, so I'm just keeping it consistent here.

But I wonder whether we even need all those barriers in this driver at all.

In that case that's fine :)

#26 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

Quoted 32 lines · Ramon Fried
Quoted 31 lines · Marek Vasut
Quoted 28 lines · Ramon Fried
Quoted 21 lines · Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +0 -0
index 7dadb10fe7..c86b9d59a5 100644

drivers/net/dwc_eth_qos.c static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length) · line 1435

Quoted 5 lines
drivers/net/dwc_eth_qos.c +2 -0
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
+       rx_desc->des0 = 0;
+       mb();

Better use wmb() for better understanding your goal here ?

Quoted 5 lines
Quoted 4 lines

The driver uses mb() all over the place, so I'm just keeping it consistent here.

But I wonder whether we even need all those barriers in this driver at all.

In that case that's fine :)

I mean, I am asking about the need for those barriers here, that's my question here.

#27 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

Quoted 34 lines · Marek Vasut
Quoted 32 lines · Ramon Fried
Quoted 31 lines · Marek Vasut
Quoted 28 lines · Ramon Fried
Quoted 21 lines · Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +0 -0
index 7dadb10fe7..c86b9d59a5 100644

drivers/net/dwc_eth_qos.c static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length) · line 1435

Quoted 5 lines
drivers/net/dwc_eth_qos.c +2 -0
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
+       rx_desc->des0 = 0;
+       mb();

Better use wmb() for better understanding your goal here ?

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

The driver uses mb() all over the place, so I'm just keeping it consistent here.

But I wonder whether we even need all those barriers in this driver at all.

In that case that's fine :)

I mean, I am asking about the need for those barriers here, that's my question here.

Better safe than sorry. unless you have the means to test it to see if it doesn't break.

Thanks,
Ramon.

#28 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

Quoted 38 lines · Ramon Fried
Quoted 37 lines · Marek Vasut
Quoted 35 lines · Ramon Fried
Quoted 34 lines · Marek Vasut
Quoted 23 lines · Ramon Fried
Quoted 22 lines · Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +0 -0
index 7dadb10fe7..c86b9d59a5 100644
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
Quoted 5 lines

drivers/net/dwc_eth_qos.c static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length) · line 1435

Quoted 4 lines
drivers/net/dwc_eth_qos.c +2 -0
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
+       rx_desc->des0 = 0;
+       mb();

Better use wmb() for better understanding your goal here ?

The driver uses mb() all over the place, so I'm just keeping it consistent here.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 1 line

But I wonder whether we even need all those barriers in this driver at all.

In that case that's fine :)

I mean, I am asking about the need for those barriers here, that's my question here.

Better safe than sorry. unless you have the means to test it to see if it doesn't break.

On STM32MP1, yes. Not on the Tegra platform.

#29 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

In reply to Marek Vasut
Quoted 33 lines · Marek Vasut

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>
---
 drivers/net/dwc_eth_qos.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/dwc_eth_qos.c +3 -2
index 7dadb10fe7..c86b9d59a5 100644
@@ -1431,8 +1431,10 @@ static int eqos_free_pkt(struct udevice *dev, uchar *packet, int length)
 
        rx_desc = &(eqos->rx_descs[eqos->rx_desc_idx]);
 
+       rx_desc->des0 = 0;
+       mb();
+       eqos->config->ops->eqos_flush_desc(rx_desc);
        eqos->config->ops->eqos_inval_buffer(packet, length);
-
        rx_desc->des0 = (u32)(ulong)packet;
        rx_desc->des1 = 0;
        rx_desc->des2 = 0;
--
Quoted 2 lines

2.25.1

#30 lore raw

PATCH 6/6 net: dwc_eth_qos: Prevent DMA from writing updated RX DMA descriptor

In reply to Marek Vasut

Dear Marek,

Quoted 7 lines

From: Marek Vasut <[email protected]>
Sent: lundi 23 mars 2020 02:45

The DMA may attempt to write a DMA descriptor in the ring while it is being updated. By writing the DMA descriptor buffer address to 0, it is assured the DMA will not use such a buffer and the buffer can be updated without any interference.

Quoted 7 lines
Cc Joe Hershberger <[email protected]>
Cc Patrice Chotard <[email protected]>
Cc Patrick Delaunay <[email protected]>
Cc Ramon Fried <[email protected]>
Cc Stephen Warren <[email protected]>

---