wic creates ext4 images that read really slow in u-boot
Took me a while to figure out why my board took 90 seconds to boot suddenly.
The issue turned out to be the ext4 partition created by wic.
On a "wic" created ext4 disk, this is what u-boot does:
ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin 19311092 bytes read in 85529 ms (219.7 KiB/s)
Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board:
ZynqMP> load mmc 0:2 0x100000 /lib/firmware/fpga.bin 19311092 bytes read in 1426 ms (12.9 MiB/s)
I checked with other files, and it turns out that ANY big file written by wic will read very slow (at 200k/s) but if I make a copy from a running system, it suddenly reads fast.
I'm not knowledgeable on ext4, so any ideas what's being passed onto the image creation tool that causes this?
Hello Mike,
sounds familiar.
Quoted 4 lines · Mike Looijmans
Took me a while to figure out why my board took 90 seconds to boot suddenly.
The issue turned out to be the ext4 partition created by wic.
I suspect it's not WIC's fault.
Quoted 10 lines
ZynqMP> load mmc 0:2
0x100000/lib/firmware/fpga.bin19311092 bytes read in 85529 ms (219.7 KiB/s)Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board:
ZynqMP> load mmc 0:2
0x10000
I'm not knowledgeable on ext4, so any ideas what's being passed onto the image
creation tool that causes this?
I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
Copying the file makes the copy being layed out in one extend probably.
I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Regards, Leon
Signature · 9 lines
Leon Woestenberg [email protected] T: +31 40 711 42 76 M: +31 6 472 30 372 Sidebranch Embedded Systems Eindhoven, The Netherlands http://www.sidebranch.com
Quoted 28 lines · Leon Woestenberg
Hello Mike,
sounds familiar.
Quoted 4 lines · Mike Looijmans
Took me a while to figure out why my board took 90 seconds to boot suddenly.
The issue turned out to be the ext4 partition created by wic.
I suspect it's not WIC's fault.
Quoted 10 lines
ZynqMP> load mmc 0:2
0x100000/lib/firmware/fpga.bin19311092 bytes read in 85529 ms (219.7 KiB/s)Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board:
ZynqMP> load mmc 0:2
0x10000
I'm not knowledgeable on ext4, so any ideas what's being passed onto the image
creation tool that causes this?
Quoted 5 lines
I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
Copying the file makes the copy being layed out in one extend probably.
If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
I am aware of two fixes for U-Boot. I will look them up, and reply again to
Quoted 14 lines
this thread.
Regards, Leon
--
Leon Woestenberg
[email protected]
T: +31 40 711 42 76
M: +31 6 472 30 372Sidebranch
Embedded Systems
Eindhoven, The Netherlands
http://www.sidebranch.com
Hello all,
Quoted 23 lines · Andre McCurdy
Quoted 22 lines · Leon Woestenberg
Hello Mike,
sounds familiar.
Quoted 3 lines · Mike Looijmans
Took me a while to figure out why my board took 90 seconds to boot suddenly. The issue turned out to be the ext4 partition created by wic.
I suspect it's not WIC's fault.
Quoted 10 lines
ZynqMP> load mmc 0:2
0x100000/lib/firmware/fpga.bin19311092 bytes read in 85529 ms (219.7 KiB/s)Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board:
ZynqMP> load mmc 0:2
0x10000
I'm not knowledgeable on ext4, so any ideas what's being passed onto the image creation tool that causes this?
Quoted 9 lines
Quoted 3 lines
I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
Copying the file makes the copy being layed out in one extend probably.
If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
I am not sure if "fragmented" is the right term in terms of filesystem details. Filesystem allocation extents (not "extends" as I mistyped earlier) are different from heavy file fragmentation. However, I agree things can be made more optimal.
So, with adding the above, there are *two* issues at play here:
1) Most/older versions of U-Boot not able to efficiently load files from ext4 filesystems, that cross multiple extents. I am aware of two fixes, see below.
2) WIC uses mkext4fs in such a way that files can cross multiple (allocation) extents. This is sub-optimal and know to cause issues with many U-Boot versions (deployed on existing systems in the field). The problems shows "randomly" with large files being deployed to the ext4 filesystem. We (OE/Yocto) may want to fix this.
Quoted 1 line
Quoted 1 line
I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Google for these two commits. I cherry-picked the first one for my project and can confirm it brings back the desired performance. (Still assuming this is the identical cause as for TO/Mike.)
commit 855b8e4f9f255415a7753819e392e4b5d984f35c
Author: Matt Madison <[email protected]> Date: Sat Aug 19 08:46:46 2017 -0700
ext4: cache extent blocks during file reads
A simpler and less-efficient approach to solving the problem with extent index blocks than what was in fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, but one that does not break write operations.
Regards,
Leon.
p.s. excuse the earlier HTML mail with signature
Quoted 67 lines · Leon Woestenberg
Hello all,
Quoted 23 lines · Andre McCurdy
Quoted 22 lines · Leon Woestenberg
Hello Mike,
sounds familiar.
Quoted 3 lines · Mike Looijmans
Took me a while to figure out why my board took 90 seconds to boot suddenly. The issue turned out to be the ext4 partition created by wic.
I suspect it's not WIC's fault.
Quoted 10 lines
ZynqMP> load mmc 0:2
0x100000/lib/firmware/fpga.bin19311092 bytes read in 85529 ms (219.7 KiB/s)Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board:
ZynqMP> load mmc 0:2
0x10000
I'm not knowledgeable on ext4, so any ideas what's being passed onto the image creation tool that causes this?Quoted 9 lines
Quoted 3 lines
I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
Copying the file makes the copy being layed out in one extend probably.
If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
I am not sure if "fragmented" is the right term in terms of filesystem details. Filesystem allocation extents (not "extends" as I mistyped earlier) are different from heavy file fragmentation. However, I agree things can be made more optimal.
So, with adding the above, there are *two* issues at play here:
1) Most/older versions of U-Boot not able to efficiently load files from ext4 filesystems, that cross multiple extents. I am aware of two fixes, see below.
2) WIC uses mkext4fs in such a way that files can cross multiple (allocation) extents. This is sub-optimal and know to cause issues with many U-Boot versions (deployed on existing systems in the field). The problems shows "randomly" with large files being deployed to the ext4 filesystem. We (OE/Yocto) may want to fix this.Quoted 1 line
Quoted 1 line
I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Google for these two commits. I cherry-picked the first one for my project and can confirm it brings back the desired performance. (Still assuming this is the identical cause as for TO/Mike.)
commit
855b8e4f9f255415a7753819e392e4b5d984f35cAuthor: Matt Madison <[email protected]> Date: Sat Aug 19 08:46:46 2017 -0700ext4: cache extent blocks during file reads
A simpler and less-efficient approach to solving the problem with extent index blocks than what was in
fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, but one that does not break write operations.
Quoted 8 lines
Signed-off-by Matt Madison <[email protected]>Regards,
Leon.
p.s. excuse the earlier HTML mail with signature
I can concur that I have also used this patch to fix speed issues with ext4 reads. The problem becomes particularly prominent when you have large files on a small partition which get over-written. I'm sure I have seen a patch go into uboot mainline recently which improved performance but I can't find it or it never actually made it into the tree.
Quoted 40 lines · Leon Woestenberg
Hello all,
Quoted 23 lines · Andre McCurdy
Quoted 22 lines · Leon Woestenberg
Hello Mike,
sounds familiar.
Quoted 3 lines · Mike Looijmans
Took me a while to figure out why my board took 90 seconds to boot suddenly. The issue turned out to be the ext4 partition created by wic.
I suspect it's not WIC's fault.
Quoted 10 lines
ZynqMP> load mmc 0:2
0x100000/lib/firmware/fpga.bin19311092 bytes read in 85529 ms (219.7 KiB/s)Now if I boot the board rename and copy that file onto itself, then it's suddenly normal again when I reboot the board:
ZynqMP> load mmc 0:2
0x10000
I'm not knowledgeable on ext4, so any ideas what's being passed onto the image creation tool that causes this?Quoted 9 lines
Quoted 3 lines
I suspect your version of U-Boot does not handle files spread across multiple filesystems (allocation) extends efficiently.
Copying the file makes the copy being layed out in one extend probably.
If WIC is generating filesystem images from scratch there's no excuse for files to be unnecessarily fragmented.
Even if some of all of the boot time can be recovered by a patch to u-boot that won't help systems which have already been deployed and don't have a way to update the bootloader.
I am not sure if "fragmented" is the right term in terms of filesystem details. Filesystem allocation extents (not "extends" as I mistyped earlier) are different from heavy file fragmentation. However, I agree things can be made more optimal.
Quoted 9 lines
So, with adding the above, there are *two* issues at play here:
1) Most/older versions of U-Boot not able to efficiently load files from ext4 filesystems, that cross multiple extents. I am aware of two fixes, see below.
2) WIC uses mkext4fs in such a way that files can cross multiple (allocation) extents. This is sub-optimal and know to cause issues with many U-Boot versions (deployed on existing systems in the field). The problems shows "randomly" with large files being deployed to the ext4 filesystem. We (OE/Yocto) may want to fix this.
That'd be best. One would expect that creating a filesystem "offline" would yield something optimal in terms of the way files are allocated, since everything about all files is already known when it's written.
My current workaround is to just mount the device and unpack the rootfs tar onto that, which yields a filesystem that u-boot can read at normal speed.
Quoted 10 lines
Quoted 1 line
Quoted 1 line
I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Google for these two commits. I cherry-picked the first one for my project and can confirm it brings back the desired performance. (Still assuming this is the identical cause as for TO/Mike.)
commit
855b8e4f9f255415a7753819e392e4b5d984f35cAuthor: Matt Madison <[email protected]> Date: Sat Aug 19 08:46:46 2017 -0700
Quoted 8 lines
ext4: cache extent blocks during file reads
A simpler and less-efficient approach to solving the problem with extent index blocks than what was in
fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, but one that does not break write operations.Signed-off-by Matt Madison <[email protected]>
I'll give it a try...
You mentioned "two" commits?
Hello Mike, all,
Quoted 35 lines · Mike Looijmans
Quoted 33 lines · Leon Woestenberg
Hello all,
Quoted 6 lines · Andre McCurdy
Quoted 5 lines · Leon Woestenberg
Hello Mike,
sounds familiar.
Quoted 8 lines
Quoted 8 lines · Mike Looijmans
Quoted 3 lines
Took me a while to figure out why my board took 90 seconds to boot suddenly. The issue turned out to be the ext4 partition created by wic.
I suspect it's not WIC's fault.
Quoted 1 line
I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Google for these two commits. I cherry-picked the first one for my project and can confirm it brings back the desired performance. (Still assuming this is the identical cause as for TO/Mike.)
commit
855b8e4f9f255415a7753819e392e4b5d984f35cAuthor: Matt Madison <[email protected]> Date: Sat Aug 19 08:46:46 2017 -0700ext4: cache extent blocks during file reads
A simpler and less-efficient approach to solving the problem with extent index blocks than what was in
fc0fc50f38a4d7d0554558076a79dfe8b0d78cd5, but one that does not break write operations.
Quoted 5 lines
Quoted 1 line
Signed-off-by Matt Madison <[email protected]>I'll give it a try...
You mentioned "two" commits?
Yes the commit above, and *another* approach referred to and mentioned in the commit message (also with hash).
The block cache feature may solve the problem in a more generic way (also caching things like directory lookups), whereas the ext4 extent patches are more specific to ext4 only.
I found the ext4 fixes starting from here:
https://github.com/madisongh/meta-tegra/issues/42
Regards,
Leon.
Quoted 3 lines · Mike Looijmans
That'd be best. One would expect that creating a filesystem "offline" would yield something optimal in terms of the way files are allocated, since everything about all files is already known when it's written.
But because we don't run image generation as root in a loopback mount, we're limited to how clever e2fsprogs's debugfs is, *not* the kernel driver.
Just a few weeks ago I fixed a bug in debugfs where it was only writing the first 2^31 bits of a file...
Ross
Quoted 1 line
Quoted 1 line
Quoted 1 line
I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Browsing through u-boot commits, I found this gem that's disabled by default:
CONFIG_BLOCK_CACHE=y
The commit is this one:
commit e40cf34a29f1b248643731a11fb1c6f0520d016c
Author: Eric Nelson <[email protected]> Date: Mon Mar 28 10:05:44 2016 -0700
drivers: block: add block device cache
Add a block device cache to speed up repeated reads of block devices by various filesystems.
This small amount of cache can dramatically speed up filesystem operations by skipping repeated reads of common areas of a block device (typically directory structures).
This has shown to have some benefit on FAT filesystem operations of loading a kernel and RAM disk, but more dramatic benefits on ext4 filesystems when the kernel and/or RAM disk are spread across multiple extent header structures as described in commit fc0fc50.
...
I'll be giving that a test drive soon...
Quoted 8 lines · Mike Looijmans
Quoted 1 line
Quoted 1 line
Quoted 1 line
I am aware of two fixes for U-Boot. I will look them up, and reply again to this thread.
Browsing through u-boot commits, I found this gem that's disabled by default:
CONFIG_BLOCK_CACHE=yThe commit is this one:
Quoted 8 lines
commit
e40cf34a29f1b248643731a11fb1c6f0520d016cAuthor: Eric Nelson <[email protected]> Date: Mon Mar 28 10:05:44 2016 -0700drivers: block: add block device cache
Add a block device cache to speed up repeated reads of block devices by various filesystems.
For the record, it's not that it's disabled by default but rather to enable it, you need to enable other "new" frameworks and so forth. And at least for "legacy" platforms, there's a certain level of pushback against doing the work required to enable that (as to be honest, it is more than just enable a few CONFIG options and re-test). A counter-pushback of "but it enables things that fix some bad performance problems!" would be really helpful.
Signature · 1 line
Tom