Skip to contentSkip to conversation

TFTP hangs with fragmented IP packets

DISCUSSION Laurent Pinchart 3 participants 5 messages Mar 29, 2022 – Apr 11, 2022

lore
Share & tools
lore
#0 lore raw

Hello,

I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.

I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.

The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.

U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.

I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.

Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.

I can test patches.

Signature · 3 lines
Regards,

Laurent Pinchart
#1 lore raw
Quoted 31 lines · Laurent Pinchart

Hello,

I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.

I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.

The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.

U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.

I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.

Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.

Quoted 6 lines

I can test patches.

--
Regards,

Laurent Pinchart

Interesting.
I will try to reproduce.
Thanks for reporting.

#2 lore raw
Quoted 36 lines · Ramon Fried
Quoted 33 lines · Laurent Pinchart

Hello,

I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.

I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.

The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.

U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.

I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.

Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.

I can test patches.

Quoted 7 lines
Quoted 4 lines

--
Regards,

Laurent Pinchart

Interesting.
I will try to reproduce.
Thanks for reporting.

Hi.
I couldn't reproduce the issue with the file you provided. I used sandbox64, and changed the TFTP BLOCK size to 4096. I managed to TFTP the file successfully.

=> setenv autoload no
=> setenv ethrotate no
=> setenv ethact eth0
=> setenv ipaddr
=> setenv serverip 172.23.1.137
=> env set ipaddr 172.23.1.100
=> tftpboot 0x100000 test.bin
Using host_eno1 device
TFTP from server 172.23.1.137; our IP address is 172.23.1.100 Filename 'test.bin'.
Load address: 0x100000
Loading: #

         718.8 KiB/s

done
Bytes transferred = 1472 (5c0 hex)
=>

Can you perhaps send me a network PCAP of your device failing to digest the file ?

Thanks,
Ramon

#3 lore raw

Hi Ramon,

Quoted 56 lines · Ramon Fried
Quoted 32 lines · Ramon Fried
Quoted 31 lines · Laurent Pinchart

Hello,

I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.

I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.

The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.

U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.

I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.

Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.

Quoted 5 lines
Quoted 2 lines

I can test patches.

Interesting.
I will try to reproduce.
Thanks for reporting.

Hi.
I couldn't reproduce the issue with the file you provided. I used sandbox64, and changed the TFTP BLOCK size to 4096. I managed to TFTP the file successfully.

=> setenv autoload no
=> setenv ethrotate no
=> setenv ethact eth0
=> setenv ipaddr
=> setenv serverip 172.23.1.137
=> env set ipaddr 172.23.1.100
=> tftpboot 0x100000 test.bin
Using host_eno1 device
TFTP from server 172.23.1.137; our IP address is 172.23.1.100 Filename 'test.bin'.
Load address: 0x100000
Loading: #

Quoted 4 lines
         718.8 KiB/s

done
Bytes transferred = 1472 (5c0 hex)
=>

Hmmmm... I wonder if it could be specific to the EQoS ethernet controller in the i.MX8, maybe caused by issues in hardware CRC calculation ? I don't have any other board I can use for testing this at the moment as I'm travelling.

Quoted 2 lines

Can you perhaps send me a network PCAP of your device failing to digest the file ?

I'll try to do that tomorrow.

Signature · 3 lines
Regards,

Laurent Pinchart
#4 lore raw
Quoted 73 lines · Laurent Pinchart

Hi Ramon,

Quoted 56 lines · Ramon Fried
Quoted 32 lines · Ramon Fried
Quoted 31 lines · Laurent Pinchart

Hello,

I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.

I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.

The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.

U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.

I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.

Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.

Quoted 5 lines
Quoted 2 lines

I can test patches.

Interesting.
I will try to reproduce.
Thanks for reporting.

Hi.
I couldn't reproduce the issue with the file you provided. I used sandbox64, and changed the TFTP BLOCK size to 4096. I managed to TFTP the file successfully.

=> setenv autoload no
=> setenv ethrotate no
=> setenv ethact eth0
=> setenv ipaddr
=> setenv serverip 172.23.1.137
=> env set ipaddr 172.23.1.100
=> tftpboot 0x100000 test.bin
Using host_eno1 device
TFTP from server 172.23.1.137; our IP address is 172.23.1.100 Filename 'test.bin'.
Load address: 0x100000
Loading: #

Quoted 4 lines
         718.8 KiB/s

done
Bytes transferred = 1472 (5c0 hex)
=>

Hmmmm... I wonder if it could be specific to the EQoS ethernet controller in the i.MX8, maybe caused by issues in hardware CRC calculation ? I don't have any other board I can use for testing this at the moment as I'm travelling.

Quoted 2 lines

Can you perhaps send me a network PCAP of your device failing to digest the file ?

Quoted 6 lines

I'll try to do that tomorrow.

--
Regards,

Laurent Pinchart

Laurent,

I'm working with a imx8mp-venice-gw74xx [1] board with EQoS<->RGMII PHY and am able to transfer your test file with CONFIG_TFTP_BLOCKSIZE=4096 fine. I haven't had any issues transferring files in general and routinely transfer files up to 513MB.

Are you running into an issue perhaps with your loadaddr position? I'm using CONFIG_SYS_LOAD_ADDR=0x40480000.

Best Regards,

Tim
[1] https://patchwork.ozlabs.org/project/uboot/list/?series=294275