U-boot: fs: add generic unaligned read offset handling
Subsystems (6)
v3 cover U-boot: fs: add generic unaligned read offset handling
[CHANGELOG]
v3:
- Fix an error that we always return 0 actread bytes for unsupported fses For unsupported fses, we should also populate @total_read. Or we will just read the data but still return 0 for actually bytes.
Now it pass all test_fs* cases.
v2
- Fix a linkage error where (U64 % U32) is called without proper helper Fix it with U64 & (U32 - 1), as the U32 value (@blocksize) should always be power of 2, thus (@blocksize - 1) is the mask we want to calculate the offset inside the block.
Above change only affects the 4th patch, everything else is not touched.
RFC->v1:
- More (manual) testing
Unfortunately, in the latest master (75967970850a), the fs-tests.sh always seems to hang at preparing the fs image.
Thus still has to do manual testing, tested btrfs, ext4 and fat, with aligned and unaligned read, also added soft link read, all looks fine here.
Extra testing is still appreciated.
- Two more btrfs specific bug fixes
All exposed during manual tests
- Remove the tailing unaligned block handling
In fact, all fses can easily handle such case, just a min() call is enough.
- Remove the support for sandboxfs
Since it's using read() calls, really no need to do block alignment check.
- Enhanced blocksize check
Ensure the returned blocksize is not only non-error, but also non-zero.
Qu Wenruo (8):
fs: fat: unexport file_fat_read_at()
fs: btrfs: fix a bug which no data get read if the length is not 0
fs: btrfs: fix a crash if specified range is beyond file size
fs: btrfs: move the unaligned read code to _fs_read() for btrfs
fs: ext4: rely on _fs_read() to handle leading unaligned block read
fs: fat: rely on higher layer to get block aligned read range
fs: ubifs: rely on higher layer to do unaligned read
fs: erofs: add unaligned read range handling
fs/btrfs/btrfs.c | 33 ++++++++--- fs/btrfs/inode.c | 89 +++-------------------------- fs/erofs/internal.h | 1 + fs/erofs/super.c | 6 ++ fs/ext4/ext4fs.c | 22 +++++++ fs/fat/fat.c | 17 +++++- fs/fs.c | 130 +++++++++++++++++++++++++++++++++++++++--- fs/ubifs/ubifs.c | 13 +++-- include/btrfs.h | 1 + include/erofs.h | 1 + include/ext4fs.h | 1 + include/fat.h | 3 +- include/ubifs_uboot.h | 1 + 13 files changed, 212 insertions(+), 106 deletions(-)
Signature · 1 line
2.37.1
PATCH v3 1/8 fs: fat: unexport file_fat_read_at()
That function is only utilized inside fat driver, unexport it.
--- fs/fat/fat.c | 4 ++-- include/fat.h | 2 -- 2 files changed, 2 insertions(+), 4 deletions(-)
fs/fat/fat.c +2 -2
@@ -1243,8 +1243,8 @@ out_free_itr:return ret; } -int file_fat_read_at (const char *filename,loff_t pos,void *buffer, -loff_t maxsize,loff_t *actread) +static int file_fat_read_at (const char *filename,loff_t pos,void *buffer, +loff_t maxsize,loff_t *actread) {fsdata fsdata;fat_itr *itr;
include/fat.h +0 -2
@@ -200,8 +200,6 @@ static inline u32 sect_to_clust(fsdata *fsdata, int sect)int file_fat_detectfs (void );int fat_exists (const char *filename);int fat_size (const char *filename,loff_t *size); -int file_fat_read_at (const char *filename,loff_t pos,void *buffer, -loff_t maxsize,loff_t *actread);int file_fat_read (const char *filename,void *buffer,int maxsize);int fat_set_blk_dev (struct blk_desc *rbdd,struct disk_partition *info);int fat_register_device (struct blk_desc *dev_desc,int part_no);
Signature · 1 line
2.37.1
PATCH v3 2/8 fs: btrfs: fix a bug which no data get read if the length is not 0
In reply to Qu Wenruo[BUG]
When testing with unaligned read, if a specific length is passed in, btrfs driver will read out nothing:
=> load host 0 $kernel_addr_r 5k_file 0x1000 0
0 bytes read in 0 ms
But if no length is passed in, it works fine, even if we pass a non-zero length:
=> load host 0 $kernel_addr_r 5k_file 0 0x1000
1024 bytes read in 0 ms
[CAUSE]
In btrfs_read() if we have a larger size than our file, we will try to truncate it using the file size.
However the real file size is not initialized if @len is not zero, thus we always truncate our length to 0, and cause the problem.
[FIX]
Fix it by just always do the file size check.
In fact btrfs_size() always follow soft link, thus it will return the real file size correctly.
--- fs/btrfs/btrfs.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
fs/btrfs/btrfs.c +9 -8
@@ -246,16 +246,17 @@ int btrfs_read(const char *file, void *buf, loff_t offset, loff_t len,return -EINVAL ; } -if (!len) { - ret =btrfs_size (file, &real_size); -if (ret <0 ) { -error ("Failed to get inode size: %s" , file); -return ret; - } - len = real_size; + ret =btrfs_size (file, &real_size); +if (ret <0 ) { +error ("Failed to get inode size: %s" , file); +return ret; } -if (len > real_size - offset) +/* +* If the length is 0 (meaning read the whole file) or the range is +* beyond file size, truncate it to the end of the file. +*/ +if (!len || len > real_size - offset) len = real_size - offset; ret =btrfs_file_read (root, ino, offset, len, buf);
Signature · 1 line
2.37.1
PATCH v3 3/8 fs: btrfs: fix a crash if specified range is beyond file size
In reply to Qu Wenruo[BUG]
When try to read a range beyond file size, btrfs driver will cause crash/segfault:
=> load host 0 $kernel_addr_r 5k_file 0 0x2000
SEGFAULT
[CAUSE]
In btrfs_read(), if @len is 0, we will truncated it to file end, but if file end is beyond our file size, this truncation will underflow @len, making it -3K in this case.
And later that @len is used to memzero the output buffer, resulting above crash.
[FIX]
Just error out if @offset is already beyond our file size.
Now it will fail properly with correct error message:
=> load host 0 $kernel_addr_r 5m_origin 0 0x2000
BTRFS: Read range beyond file size, offset 8192 file size 5120
Failed to load '5m_origin'
--- fs/btrfs/btrfs.c | 6 ++++++ 1 file changed, 6 insertions(+)
fs/btrfs/btrfs.c +6 -0
@@ -252,6 +252,12 @@ int btrfs_read(const char *file, void *buf, loff_t offset, loff_t len,return ret; } +if (offset >= real_size) { +error ("Read range beyond file size, offset %llu file size %llu" , + offset, real_size); +return -EINVAL ; + } +/* * If the length is 0 (meaning read the whole file) or the range is * beyond file size, truncate it to the end of the file.
Signature · 1 line
2.37.1
PATCH v3 4/8 fs: btrfs: move the unaligned read code to _fs_read() for btrfs
In reply to Qu WenruoUnlike FUSE or kernel, U-boot filesystem code makes the underly fs code to handle the unaligned read (aka, read range is not aligned to fs block size).
This makes underlying fs code harder to implement, as they have to handle unaligned read all by themselves.
This patch will change the behavior, starting from btrfs, by moving the unaligned read code into _fs_read().
The idea is pretty simple, if we have an unaligned read request, we handle it in the following steps:
1. Grab the blocksize of the fs
2. Read the leading unaligned range
We will read the block that @offset is in, and copy the requested part into buf.
The the block we read covers the whole range, we just call it a day.
3. Read the remaining part
The tailing part may be unaligned, but all fses handles the tailing part much easier than the leading unaligned part.
As they just need to do a min(extent_size, start + len - cur) to calculate the real read size.
In fact, for most file reading, the file size is not aligned and we need to handle the tailing part anyway.
There is a btrfs specific cleanup involved:
- In btrfs_file_read(), merge the tailing unaligned read into the main loop.
Just reuse the existing read length calculation is enough.
- Remove read_and_truncate_page() call
Since there is no explicit leading/tailing unaligned read anymore.
This has been tested with a proper randomly populated btrfs file, then tried in sandbox mode with different aligned and unaligned range and compare the output with md5sum.
--- fs/btrfs/btrfs.c | 10 ++++ fs/btrfs/inode.c | 89 +++----------------------------- fs/fs.c | 130 ++++++++++++++++++++++++++++++++++++++++++++--- include/btrfs.h | 1 + 4 files changed, 141 insertions(+), 89 deletions(-)
fs/btrfs/btrfs.c +10 -0
@@ -234,6 +234,10 @@ int btrfs_read(const char *file, void *buf, loff_t offset, loff_t len,int ret;ASSERT (fs_info); + +/* Higher layer has ensures it never pass unaligned offset in. */ +ASSERT (IS_ALIGNED (offset, fs_info->sectorsize)); + ret =btrfs_lookup_path (fs_info->fs_root,BTRFS_FIRST_FREE_OBJECTID , file, &root, &ino, &type,40 );if (ret <0 ) {
@@ -275,6 +279,12 @@ int btrfs_read(const char *file, void *buf, loff_t offset, loff_t len,return 0 ; } +int btrfs_get_blocksize (const char *filename) +{ +ASSERT (current_fs_info); +return current_fs_info->sectorsize; +} +void btrfs_close (void ) {if (current_fs_info) {
fs/btrfs/inode.c +8 -81
120 lines not on this page · raw message
fs/fs.c +122 -8
220 lines not on this page · raw message
include/btrfs.h +1 -0
@@ -17,6 +17,7 @@ int btrfs_ls(const char *);int btrfs_exists (const char *);int btrfs_size (const char *,loff_t *);int btrfs_read (const char *,void *,loff_t ,loff_t ,loff_t *); +int btrfs_get_blocksize (const char *);void btrfs_close (void );int btrfs_uuid (char *);void btrfs_list_subvols (void );
Signature · 1 line
2.37.1
PATCH v3 5/8 fs: ext4: rely on _fs_read() to handle leading unaligned block read
In reply to Qu WenruoJust add ext4_get_blocksize() and a new assert() in ext4fs_read_file().
---
Several cleanup candicate:
1. ext_fs->blksz is never populated, thus it's always 0 We can not easily grab blocksize just like btrfs in this case.
Thus we have to go the same calculation in ext4fs_read_file().
2. can not easily cleanup @skipfirst in ext4fs_read_file() It seems to screw up with soft link read.
--- fs/ext4/ext4fs.c | 22 ++++++++++++++++++++++ fs/fs.c | 2 +- include/ext4fs.h | 1 + 3 files changed, 24 insertions(+), 1 deletion(-)
fs/ext4/ext4fs.c +22 -0
@@ -69,6 +69,8 @@ int ext4fs_read_file(struct ext2fs_node *node, loff_t pos,short status;struct ext_block_cache cache; +assert (IS_ALIGNED (pos, blocksize)); +ext_cache_init (&cache);/* Adjust len so it we can't read past the end of the file. */
@@ -259,6 +261,26 @@ int ext4_read_file(const char *filename, void *buf, loff_t offset, loff_t len,return ext4fs_read (buf, offset, len, len_read); } +int ext4_get_blocksize (const char *filename) +{ +struct ext_filesystem *fs; +int log2blksz; +int log2_fs_blocksize; +loff_t file_len; +int ret; + + ret =ext4fs_open (filename, &file_len); +if (ret <0 ) { +printf ("** File not found %s **\n" , filename); +return -1 ; + } + fs =get_fs (); + log2blksz = fs->dev_desc->log2blksz; + log2_fs_blocksize =LOG2_BLOCK_SIZE (ext4fs_file->data) - log2blksz; + +return (1 << (log2_fs_blocksize + log2blksz)); +} +int ext4fs_uuid (char *uuid_str) {if (ext4fs_root ==NULL )
fs/fs.c +1 -1
@@ -237,7 +237,7 @@ static struct fstype_info fstypes[] = { .exists = ext4fs_exists, .size = ext4fs_size, .read = ext4_read_file, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = ext4_get_blocksize, #ifdef CONFIG_CMD_EXT4_WRITE .write = ext4_write_file, .ln = ext4fs_create_link,
include/ext4fs.h +1 -0
@@ -161,6 +161,7 @@ int ext4fs_probe(struct blk_desc *fs_dev_desc, struct disk_partition *fs_partition);int ext4_read_file (const char *filename,void *buf,loff_t offset,loff_t len,loff_t *actread); +int ext4_get_blocksize (const char *filename);int ext4_read_superblock (char *buffer);int ext4fs_uuid (char *uuid_str);void ext_cache_init (struct ext_block_cache *cache);
Signature · 1 line
2.37.1
PATCH v3 6/8 fs: fat: rely on higher layer to get block aligned read range
In reply to Qu WenruoJust implement fat_get_blocksize() for fat, so that fat_read_file() always get a block aligned read range.
Unfortunately I'm not experienced enough to cleanup the fat code, thus further cleanup is appreciated.
--- fs/fat/fat.c | 13 +++++++++++++ fs/fs.c | 2 +- include/fat.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)
fs/fat/fat.c +13 -0
@@ -1299,6 +1299,19 @@ int fat_read_file(const char *filename, void *buf, loff_t offset, loff_t len,return ret; } +int fat_get_blocksize (const char *filename) +{ +fsdata fsdata = {0 }; +int ret; + + ret =get_fs_info (&fsdata); +if (ret) +return ret; + +free (fsdata.fatbuf); +return fsdata.sect_size; +} + typedef struct { struct fs_dir_stream parent; struct fs_dirent dirent;
fs/fs.c +1 -1
@@ -208,7 +208,7 @@ static struct fstype_info fstypes[] = { .exists = fat_exists, .size = fat_size, .read = fat_read_file, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = fat_get_blocksize,#if CONFIG_IS_ENABLED (FAT_WRITE ) .write = file_fat_write, .unlink = fat_unlink,
include/fat.h +1 -0
@@ -201,6 +201,7 @@ int file_fat_detectfs(void);int fat_exists (const char *filename);int fat_size (const char *filename,loff_t *size);int file_fat_read (const char *filename,void *buffer,int maxsize); +int fat_get_blocksize (const char *filename);int fat_set_blk_dev (struct blk_desc *rbdd,struct disk_partition *info);int fat_register_device (struct blk_desc *dev_desc,int part_no);
Signature · 1 line
2.37.1
PATCH v3 7/8 fs: ubifs: rely on higher layer to do unaligned read
In reply to Qu WenruoCurrently ubifs doesn't support unaligned read offset, thanks to the recent _fs_read() work to handle unaligned read, we only need to implement ubifs_get_blocksize() to take advantage of it.
Now ubifs can do unaligned read without any problem.
---
Unfortunately I can not test ubifs, as enabling UBI would cause compile failure due to missing of <asm/atomic.h> header.
--- fs/fs.c | 2 +- fs/ubifs/ubifs.c | 13 ++++++++----- include/ubifs_uboot.h | 1 + 3 files changed, 10 insertions(+), 6 deletions(-)
fs/fs.c +1 -1
@@ -312,7 +312,7 @@ static struct fstype_info fstypes[] = { .exists = ubifs_exists, .size = ubifs_size, .read = ubifs_read, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = ubifs_get_blocksize, .write = fs_write_unsupported, .uuid = fs_uuid_unsupported, .opendir = fs_opendir_unsupported,
fs/ubifs/ubifs.c +8 -5
@@ -846,11 +846,9 @@ int ubifs_read(const char *filename, void *buf, loff_t offset, *actread =0 ; -if (offset & (PAGE_SIZE -1 )) { -printf ("ubifs: Error offset must be a multiple of %d\n" , -PAGE_SIZE ); -return -1 ; - } +/* Higher layer should ensure it always pass page aligned range. */ +assert (IS_ALIGNED (offset,PAGE_SIZE )); +assert (IS_ALIGNED (size,PAGE_SIZE )); c->ubi =ubi_open_volume (c->vi.ubi_num, c->vi.vol_id,UBI_READONLY );/* ubifs_findfile will resolve symlinks, so we know that we get
@@ -920,6 +918,11 @@ out:return err; } +int ubifs_get_blocksize (const char *filename) +{ +return PAGE_SIZE ; +} +void ubifs_close (void ) { }
include/ubifs_uboot.h +1 -0
@@ -29,6 +29,7 @@ int ubifs_exists(const char *filename);int ubifs_size (const char *filename,loff_t *size);int ubifs_read (const char *filename,void *buf,loff_t offset,loff_t size,loff_t *actread); +int ubifs_get_blocksize (const char *filename);void ubifs_close (void );#endif /* __UBIFS_UBOOT_H__ */
Signature · 1 line
2.37.1
PATCH v3 8/8 fs: erofs: add unaligned read range handling
In reply to Qu WenruoI'm not an expert on erofs, but my quick glance didn't expose any special handling on unaligned range, thus I think the U-boot erofs driver doesn't really support unaligned read range.
This patch will add erofs_get_blocksize() so erofs can benefit from the generic unaligned read support.
--- fs/erofs/internal.h | 1 + fs/erofs/super.c | 6 ++++++ fs/fs.c | 2 +- include/erofs.h | 1 + 4 files changed, 9 insertions(+), 1 deletion(-)
fs/erofs/internal.h +1 -0
@@ -83,6 +83,7 @@ struct erofs_sb_info {u16 available_compr_algs;u16 lz4_max_distance;u32 checksum; +u32 blocksize;u16 extra_devices;union {u16 devt_slotoff;/* used for mkfs */
fs/erofs/super.c +6 -0
@@ -99,7 +99,13 @@ int erofs_read_superblock(void) sbi.build_time =le64_to_cpu (dsb->build_time); sbi.build_time_nsec =le32_to_cpu (dsb->build_time_nsec); + sbi.blocksize =1 << blkszbits;memcpy (&sbi.uuid, dsb->uuid,sizeof (dsb->uuid));return erofs_init_devices (&sbi, dsb); } + +int erofs_get_blocksize (const char *filename) +{ +return sbi.blocksize; +}
fs/fs.c +1 -1
@@ -376,7 +376,7 @@ static struct fstype_info fstypes[] = { .readdir = erofs_readdir, .ls = fs_ls_generic, .read = erofs_read, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = erofs_get_blocksize, .size = erofs_size, .close = erofs_close, .closedir = erofs_closedir,
include/erofs.h +1 -0
@@ -10,6 +10,7 @@ int erofs_probe(struct blk_desc *fs_dev_desc, struct disk_partition *fs_partition);int erofs_read (const char *filename,void *buf,loff_t offset,loff_t len,loff_t *actread); +int erofs_get_blocksize (const char *filename);int erofs_size (const char *filename,loff_t *size);int erofs_exists (const char *filename);void erofs_close (void );
Signature · 1 line
2.37.1
Quoted 7 lines · Qu Wenruo
[CHANGELOG]
v3:
- Fix an error that we always return 0 actread bytes for unsupported fses For unsupported fses, we should also populate @total_read. Or we will just read the data but still return 0 for actually bytes.Now it pass all test_fs* cases.
We're failing squashfs still, sorry:
https://source.denx.de/u-boot/u-boot/-/jobs/486819#L399
Signature · 1 line
Tom