net: stmmac: Set OWN bit last in dwmac4_set_rx_owner()
Subsystems (4)
PATCH 1/1 net: stmmac: Set OWN bit last in dwmac4_set_rx_owner()
Ensure that all other bits in the RDES3 descriptor are configured before transferring ownership of the descriptor to DMA via the OWN bit.
--- drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c +4 -1
@@ -186,10 +186,13 @@ static void dwmac4_set_tx_owner(struct dma_desc *p)static void dwmac4_set_rx_owner (struct dma_desc *p,int disable_rx_ic) { - p->des3 |=cpu_to_le32 (RDES3_OWN |RDES3_BUFFER1_VALID_ADDR ); + p->des3 |=cpu_to_le32 (RDES3_BUFFER1_VALID_ADDR );if (!disable_rx_ic) p->des3 |=cpu_to_le32 (RDES3_INT_ON_COMPLETION_EN ); + +dma_wmb (); + p->des3 |=cpu_to_le32 (RDES3_OWN ); }static int dwmac4_get_tx_ls (struct dma_desc *p)
Signature · 1 line
2.34.1
Quoted 2 lines · Tan En De
Ensure that all other bits in the RDES3 descriptor are configured before transferring ownership of the descriptor to DMA via the OWN bit.
Are you seeing things going wrong with real hardware, or is this just code review? If this is a real problem, please add a description of what the user would see.
Does this need to be backported in stable?
https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
If it does, you should add a Fixes: tag and 'Cc: [email protected]'
This will also decide which tree you need to base the patch on:
https://www.kernel.org/doc/html/latest/process/maintainer-netdev.html#netdev-faq
drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c static void dwmac4_set_tx_owner(struct dma_desc *p) · line 195
Quoted 10 lines
static void
dwmac4_set_rx_owner(struct dma_desc *p, int disable_rx_ic){drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c +4
-1- p->des3 |=cpu_to_le32 (RDES3_OWN |RDES3_BUFFER1_VALID_ADDR ); + p->des3 |=cpu_to_le32 (RDES3_BUFFER1_VALID_ADDR );if (!disable_rx_ic) p->des3 |=cpu_to_le32 (RDES3_INT_ON_COMPLETION_EN ); + +dma_wmb (); + p->des3 |=cpu_to_le32 (RDES3_OWN );
Is the problem here that RDES3_INT_ON_COMPLETION_EN is added after the RDES3_OWN above has hit the hardware, so it gets ignored?
It seems like it would be better to calculate the value in a local variable, and then assign to p->des3 once.
Andrew
Hi Andrew, thanks for taking a look at my patch.
Quoted 5 lines
Are you seeing things going wrong with real hardware, or is this just code review? If this is a real problem, please add a description of what the user would see.
Does this need to be backported in stable?
On my FPGA, after running iperf3 for a while, the GMAC somehow got stuck, as shown by 0.00 bits/sec, for example:
``` iperf3 -t 6000 -c 192.168.xxx.xxx -i 10 -P 2 -l 128 ... [ 5] 220.00-230.00 sec 2.04 MBytes 1.71 Mbits/sec 3 1.41 KBytes [ 7] 220.00-230.00 sec 2.04 MBytes 1.71 Mbits/sec 3 1.41 KBytes [SUM] 220.00-230.00 sec 4.07 MBytes 3.42 Mbits/sec 6 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 230.00-240.01 sec 0.00 Bytes 0.00 bits/sec 2 1.41 KBytes [ 7] 230.00-240.01 sec 0.00 Bytes 0.00 bits/sec 2 1.41 KBytes [SUM] 230.00-240.01 sec 0.00 Bytes 0.00 bits/sec 4 ```
Used devmem to check registers:0x780 (Rx_Packets_Count_Good_Bad)
- The count was incrementing, so packets were still received. 0x114C (DMA_CH0_Current_App_RxDesc)
- Value was changing, so DMA did update the RX descriptor address pointer. 0x1160 (DMA_CH0_Status).
- Receive Buffer Unavailable RBU = 0.
- Receive Interrupt RI = 1 (stuck at 1).
which led me to suspect that there was missed RX interrupt.
I then came across dwmac4_set_rx_owner() function, and saw the possibility of missed interrupt (when DMA sees OWN bit before INT_ON_COMPLETION bit).
However, even with this patch, the problem persists on my FPGA. Therefore, I'd treat this patch as a code review, as I can't provide a concrete proof that this patch fixes any real hardware.
Quoted 5 lines
Is the problem here that
RDES3_INT_ON_COMPLETION_ENis added after theRDES3_OWNabove has hit the hardware, so it gets ignored?It seems like it would be better to calculate the value in a local variable, and then assign to
p->des3once.
I didn't use local variable because I worry about CPU out-of-order execution. For example,
``` local_var = (INT_ON_COMPLETION | OWN) des3 |= local_var ``` CPU optimization might result in this ``` des3 |= INT_ON_COMPLETION des3 |= OWN ``` or worst, out of order like this ``` des3 |= OWN des3 |= INT_ON_COMPLETION ```
which could cause missing interrupt.
Regards,
Tan En De
Quoted 31 lines · EnDe Tan
Hi Andrew, thanks for taking a look at my patch.
Quoted 5 lines
Are you seeing things going wrong with real hardware, or is this just code review? If this is a real problem, please add a description of what the user would see.
Does this need to be backported in stable?
On my FPGA, after running iperf3 for a while, the GMAC somehow got stuck, as shown by 0.00 bits/sec, for example:
``` iperf3 -t 6000 -c 192.168.xxx.xxx -i 10 -P 2 -l 128 ... [ 5] 220.00-230.00 sec 2.04 MBytes 1.71 Mbits/sec 3 1.41 KBytes [ 7] 220.00-230.00 sec 2.04 MBytes 1.71 Mbits/sec 3 1.41 KBytes [SUM] 220.00-230.00 sec 4.07 MBytes 3.42 Mbits/sec 6 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 230.00-240.01 sec 0.00 Bytes 0.00 bits/sec 2 1.41 KBytes [ 7] 230.00-240.01 sec 0.00 Bytes 0.00 bits/sec 2 1.41 KBytes [SUM] 230.00-240.01 sec 0.00 Bytes 0.00 bits/sec 4 ```Used devmem to check registers:
0x780(Rx_Packets_Count_Good_Bad)
- The count was incrementing, so packets were still received.0x114C(DMA_CH0_Current_App_RxDesc)
- Value was changing, so DMA did update the RX descriptor address pointer.0x1160(DMA_CH0_Status).
- Receive Buffer Unavailable RBU = 0.
- Receive Interrupt RI = 1 (stuck at 1).
Quoted 7 lines
which led me to suspect that there was missed RX interrupt.
I then came acrossdwmac4_set_rx_owner()function, and saw the possibility of missed interrupt (when DMA sees OWN bit beforeINT_ON_COMPLETIONbit).However, even with this patch, the problem persists on my FPGA. Therefore, I'd treat this patch as a code review, as I can't provide a concrete proof that this patch fixes any real hardware.
O.K. Please target this patch to net-next. We can always get it backported to stable later.
Quoted 22 lines
Quoted 5 lines
Is the problem here that
RDES3_INT_ON_COMPLETION_ENis added after theRDES3_OWNabove has hit the hardware, so it gets ignored?It seems like it would be better to calculate the value in a local variable, and then assign to
p->des3once.I didn't use local variable because I worry about CPU out-of-order execution. For example,
``` local_var = (INT_ON_COMPLETION | OWN) des3 |= local_var ``` CPU optimization might result in this ``` des3 |= INT_ON_COMPLETION des3 |= OWN ``` or worst, out of order like this ``` des3 |= OWN des3 |= INT_ON_COMPLETION ```
Quoted 1 line
which could cause missing interrupt.
I'm assuming the des3 is mapped as non-cached. So each access is expensive. If you can convince the compiler to assemble the value in a register and then perform one write, it will be cheaper than two writes.
Andrew