TFTP hangs with fragmented IP packets
Hello,
I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.
I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.
The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.
U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.
I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.
Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.
I can test patches.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 31 lines · Laurent Pinchart
Hello,
I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.
I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.
The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.
U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.
I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.
Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.
Quoted 6 lines
I can test patches.
--
Regards,Laurent Pinchart
Interesting.
I will try to reproduce.
Thanks for reporting.
Quoted 36 lines · Ramon Fried
Quoted 33 lines · Laurent Pinchart
Hello,
I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.
I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.
The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.
U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.
I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.
Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.
I can test patches.
Quoted 7 lines
Quoted 4 lines
--
Regards,Laurent Pinchart
Interesting.
I will try to reproduce.
Thanks for reporting.
Hi.
I couldn't reproduce the issue with the file you provided. I used sandbox64, and changed the TFTP BLOCK size to 4096. I managed to TFTP the file successfully.
=> setenv autoload no
=> setenv ethrotate no
=> setenv ethact eth0
=> setenv ipaddr
=> setenv serverip 172.23.1.137
=> env set ipaddr 172.23.1.100
=> tftpboot 0x100000 test.bin
Using host_eno1 device
TFTP from server 172.23.1.137; our IP address is 172.23.1.100 Filename 'test.bin'.
Load address: 0x100000
Loading: #
718.8 KiB/s
done
Bytes transferred = 1472 (5c0 hex)
=>
Can you perhaps send me a network PCAP of your device failing to digest the file ?
Thanks,
Ramon
Hi Ramon,
Quoted 56 lines · Ramon Fried
Quoted 32 lines · Ramon Fried
Quoted 31 lines · Laurent Pinchart
Hello,
I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.
I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.
The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.
U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.
I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.
Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.
Quoted 5 lines
Quoted 2 lines
I can test patches.
Interesting.
I will try to reproduce.
Thanks for reporting.Hi.
I couldn't reproduce the issue with the file you provided. I used sandbox64, and changed the TFTP BLOCK size to 4096. I managed to TFTP the file successfully.=> setenv autoload no
=> setenv ethrotate no
=> setenv ethact eth0
=> setenv ipaddr
=> setenv serverip 172.23.1.137
=> env set ipaddr 172.23.1.100
=> tftpboot0x100000test.bin
Usinghost_eno1device
TFTP from server 172.23.1.137; our IP address is 172.23.1.100 Filename 'test.bin'.
Load address:0x100000
Loading: #
Quoted 4 lines
718.8 KiB/sdone
Bytes transferred = 1472 (5c0 hex)
=>
Hmmmm... I wonder if it could be specific to the EQoS ethernet controller in the i.MX8, maybe caused by issues in hardware CRC calculation ? I don't have any other board I can use for testing this at the moment as I'm travelling.
Quoted 2 lines
Can you perhaps send me a network PCAP of your device failing to digest the file ?
I'll try to do that tomorrow.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 73 lines · Laurent Pinchart
Hi Ramon,
Quoted 56 lines · Ramon Fried
Quoted 32 lines · Ramon Fried
Quoted 31 lines · Laurent Pinchart
Hello,
I've banged my head a few days ago trying to debug an issue with a TFTP transfer hanging in the middle.
I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and the board uses the EQoS ethernet controller.
The problem started occurring after rebuilding a kernel image. U-Boot started transferring the image, and stopped in the middle, eventually timing out. Capture network traffic showed that U-Boot was continuously asking for retransmit of the same block, and eventually timed out.
U-Boot is configured with a default TFTP block size of 4096 bytes, which results in the TFTP blocks being sent in one UDP packet split in three IP packets. U-Boot is configured with IP fragmentation supprot enabled. This works fine for all TFTP blocks until a paticular one in the middle of the kernel image.
I've narrowed it down to a file of 1472 that can't be transferred at all (I have attached the binary to this e-mail). Changing the value of any of the last two bytes of the file allows transferring it correctly, so I suspect a CRC issue, likely related to IP fragmentation. Lowering the TFTP block size to avoid fragmentation works around the problem.
Arguably a TFTP block size of 4096 bytes should probably not be used with a 1500 bytes MTU network, but I thought it would be useful to fix the issue nonetheless.
Quoted 5 lines
Quoted 2 lines
I can test patches.
Interesting.
I will try to reproduce.
Thanks for reporting.Hi.
I couldn't reproduce the issue with the file you provided. I used sandbox64, and changed the TFTP BLOCK size to 4096. I managed to TFTP the file successfully.=> setenv autoload no
=> setenv ethrotate no
=> setenv ethact eth0
=> setenv ipaddr
=> setenv serverip 172.23.1.137
=> env set ipaddr 172.23.1.100
=> tftpboot0x100000test.bin
Usinghost_eno1device
TFTP from server 172.23.1.137; our IP address is 172.23.1.100 Filename 'test.bin'.
Load address:0x100000
Loading: #Quoted 4 lines
718.8 KiB/sdone
Bytes transferred = 1472 (5c0 hex)
=>Hmmmm... I wonder if it could be specific to the EQoS ethernet controller in the i.MX8, maybe caused by issues in hardware CRC calculation ? I don't have any other board I can use for testing this at the moment as I'm travelling.
Quoted 2 lines
Can you perhaps send me a network PCAP of your device failing to digest the file ?
Quoted 6 lines
I'll try to do that tomorrow.
--
Regards,Laurent Pinchart
Laurent,
I'm working with a imx8mp-venice-gw74xx [1] board with EQoS<->RGMII PHY and am able to transfer your test file with CONFIG_TFTP_BLOCKSIZE=4096 fine. I haven't had any issues transferring files in general and routinely transfer files up to 513MB.
Are you running into an issue perhaps with your loadaddr position? I'm using CONFIG_SYS_LOAD_ADDR=0x40480000.
Best Regards,
Tim
[1] https://patchwork.ozlabs.org/project/uboot/list/?series=294275