u-boot: fs: add generic unaligned read handling
Subsystems (6)
cover u-boot: fs: add generic unaligned read handling
[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.
Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).
But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.
[ADVANTAGE]
This patchset will handle unaligned read range in _fs_read():
- Get blocksize of the underlying fs
- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.
If the first block covers the whole range, we just call it aday.
- Read the aligned range if there is any
- Read the tailing block containing the unaligned range And copy the covered range into the destination.
[DISADVANTAGE]
There are mainly two problems:
- Extra memory allocation for every _fs_read() call For the leading and tailing block.
- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.
[SUPPORTED FSES]
- Btrfs (manually tested*)
- Ext4 (manually tested)
- FAT (manually tested)
- Erofs
- sandboxfs
- ubifs
*: Failed to get the test cases run, thus have to go sandbox mode, and attach an image with target fs, load the target file (with unaligned range) and compare the result using md5sum.
For EXT4/FAT, they may need extra cleanup, as their existing unaligned range handling is no longer needed anymore, cleaning them up should free more code lines than the added one.
Just not confident enough to modify them all by myself.
[UNSUPPORTED FSES]
- Squashfs
They don't support non-zero offset, thus it can not handle the tailing block reading.
Need extra help to add block aligned offset support.
- Semihostfs
It's using hardcoded trap to do system calls, not sure how it would work for stat() call.
Extra testing/feedback is always appreciated.
Qu Wenruo (8):
fs: fat: unexport file_fat_read_at()
fs: always get the file size in _fs_read()
fs: btrfs: move the unaligned read code to _fs_read() for btrfs
fs: ext4: rely on _fs_read() to pass block aligned range into
ext4fs_read_file()
fs: fat: rely on higher layer to get block aligned read range
fs: sandboxfs: add sandbox_fs_get_blocksize()
fs: ubifs: rely on higher layer to do unaligned read
fs: erofs: add unaligned read range handling
arch/sandbox/cpu/os.c | 11 +++ fs/btrfs/btrfs.c | 24 +++--- fs/btrfs/inode.c | 84 ++------------------ fs/erofs/internal.h | 1 + fs/erofs/super.c | 6 ++ fs/ext4/ext4fs.c | 11 +++ fs/fat/fat.c | 17 ++++- fs/fs.c | 169 +++++++++++++++++++++++++++++++++++++++-- fs/sandbox/sandboxfs.c | 14 ++++ fs/ubifs/ubifs.c | 13 ++-- include/btrfs.h | 1 + include/erofs.h | 1 + include/ext4fs.h | 1 + include/fat.h | 3 +- include/os.h | 8 ++ include/sandboxfs.h | 1 + include/ubifs_uboot.h | 1 + 17 files changed, 258 insertions(+), 108 deletions(-)
Signature · 1 line
2.36.1
RFC PATCH 1/8 fs: fat: unexport file_fat_read_at()
That function is only utilized inside fat driver, unexport it.
--- fs/fat/fat.c | 4 ++-- include/fat.h | 2 -- 2 files changed, 2 insertions(+), 4 deletions(-)
fs/fat/fat.c +2 -2
@@ -1243,8 +1243,8 @@ out_free_itr:return ret; } -int file_fat_read_at (const char *filename,loff_t pos,void *buffer, -loff_t maxsize,loff_t *actread) +static int file_fat_read_at (const char *filename,loff_t pos,void *buffer, +loff_t maxsize,loff_t *actread) {fsdata fsdata;fat_itr *itr;
include/fat.h +0 -2
@@ -200,8 +200,6 @@ static inline u32 sect_to_clust(fsdata *fsdata, int sect)int file_fat_detectfs (void );int fat_exists (const char *filename);int fat_size (const char *filename,loff_t *size); -int file_fat_read_at (const char *filename,loff_t pos,void *buffer, -loff_t maxsize,loff_t *actread);int file_fat_read (const char *filename,void *buffer,int maxsize);int fat_set_blk_dev (struct blk_desc *rbdd,struct disk_partition *info);int fat_register_device (struct blk_desc *dev_desc,int part_no);
Signature · 1 line
2.36.1
Re: [PATCH RFC 1/8] fs: fat: unexport file_fat_read_at()
Quoted 8 lines · Qu Wenruo
That function is only utilized inside fat driver, unexport it.
Signed-off-by Qu Wenruo <[email protected]>--- fs/fat/fat.c | 4 ++-- include/fat.h | 2 -- 2 files changed, 2 insertions(+), 4 deletions(-)
Quoted 32 lines
fs/fat/fat.c +2
-2@@ -1243,8 +1243,8 @@ out_free_itr:return ret; } -int file_fat_read_at (const char *filename,loff_t pos,void *buffer, -loff_t maxsize,loff_t *actread) +static int file_fat_read_at (const char *filename,loff_t pos,void *buffer, +loff_t maxsize,loff_t *actread) {fsdata fsdata;fat_itr *itr;include/fat.h +0
-3@@ -200,8 +200,6 @@ static inline u32 sect_to_clust(fsdata *fsdata, int sect)int file_fat_detectfs (void );int fat_exists (const char *filename);int fat_size (const char *filename,loff_t *size); -int file_fat_read_at (const char *filename,loff_t pos,void *buffer, -loff_t maxsize,loff_t *actread);int file_fat_read (const char *filename,void *buffer,int maxsize);int fat_set_blk_dev (struct blk_desc *rbdd,struct disk_partition *info);int fat_register_device (struct blk_desc *dev_desc,int part_no); --2.36.1
RFC PATCH 2/8 fs: always get the file size in _fs_read()
In reply to Qu WenruoFor _fs_read(), @len == 0 means we read the whole file. And we just pass @len == 0 to make each filesystem to handle it.
In fact we have info->size() call to properly get the filesize.
So we can not only call info->size() to grab the file_size for len == 0 case, but also detect invalid @len (e.g. @len > file_size) in advance or truncate @len.
This behavior also allows us to handle unaligned better in the incoming patches.
--- fs/fs.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)
fs/fs.c +21 -4
@@ -579,6 +579,7 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, {struct fstype_info *info =fs_get_info (fs_type);void *buf; +loff_t file_size;int ret;#ifdef CONFIG_LMB
@@ -589,10 +590,26 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, }#endif -/* -* We don't actually know how many bytes are being read, since len==0 -* means read the whole file. -*/ + ret = info->size (filename, &file_size); +if (ret <0 ) { +log_err ("** Unable to get file size for %s, %d **\n" , + filename, ret); +return ret; + } +if (offset >= file_size) { +log_err ( +"** Invalid offset, offset (%llu) >= file size (%llu)\n" , + offset, file_size); +return -EINVAL ; + + } +if (len ==0 || offset + len > file_size) { +if (len > file_size) +log_info ( +"** Truncate read length from %llu to %llu, as file size is %llu **\n" , + len, file_size, file_size); + len = file_size - offset; + } buf =map_sysmem (addr, len); ret = info->read (filename, buf, offset, len, actread);unmap_sysmem (buf);
Signature · 1 line
2.36.1
Re: [PATCH RFC 2/8] fs: always get the file size in _fs_read()
Hi, wenruo,
Quoted 30 lines · Qu Wenruo
For
_fs_read(), @len == 0 means we read the whole file. And we just pass @len == 0 to make each filesystem to handle it.In fact we have
info->size()call to properly get the filesize.So we can not only call
info->size()to grab thefile_sizefor len == 0 case, but also detect invalid @len (e.g. @len >file_size) in advance or truncate @len.This behavior also allows us to handle unaligned better in the incoming patches.
Signed-off-by Qu Wenruo <[email protected]>--- fs/fs.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)fs/fs.c +1
-0@@ -579,6 +579,7 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, {struct fstype_info *info =fs_get_info (fs_type);void *buf; +loff_t file_size;int ret;#ifdef CONFIG_LMB
fs/fs.c static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, · line 593
Quoted 8 lines
fs/fs.c +1
-4@@ -589,10 +590,26 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, }#endif -/* -* We don't actually know how many bytes are being read, since len==0 -* means read the whole file. -*/ + ret = info->size (filename, &file_size);
I get an error when running the erofs test cases. The return value isn't as expected when reading symlink file.
For symlink file, erofs_size will return the size of the symlink itself here.
Quoted 8 lines
fs/fs.c +8
-0+if (ret <0 ) { +log_err ("** Unable to get file size for %s, %d **\n" , + filename, ret); +return ret; + } +if (offset >= file_size) { + log_err( + "** Invalid offset, offset (%llu) >= file size (%llu)\n",
fs/fs.c static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, · line 611
Quoted 10 lines
fs/fs.c +10
-0+ offset, file_size); +return -EINVAL ; + + } +if (len ==0 || offset + len > file_size) { +if (len > file_size) +log_info ( +"** Truncate read length from %llu to %llu, as file size is %llu **\n" , + len, file_size, file_size); + len = file_size - offset;
Then, we will get a wrong len in the case of len==0. So I think we need to do something extra with the symlink file?
Thanks,
Jianan
Quoted 4 lines
fs/fs.c +1
-0+ } buf =map_sysmem (addr, len); ret = info->read (filename, buf, offset, len, actread);unmap_sysmem (buf);
Re: [PATCH RFC 2/8] fs: always get the file size in _fs_read()
Quoted 40 lines · Huang Jianan
Hi, wenruo,
Quoted 37 lines · Qu Wenruo
For
_fs_read(), @len == 0 means we read the whole file. And we just pass @len == 0 to make each filesystem to handle it.In fact we have
info->size()call to properly get the filesize.So we can not only call
info->size()to grab thefile_sizefor len == 0 case, but also detect invalid @len (e.g. @len >file_size) in advance or truncate @len.This behavior also allows us to handle unaligned better in the incoming patches.
Signed-off-by Qu Wenruo <[email protected]>---
fs/fs.c| 25 +++++++++++++++++++++----1 file changed, 21 insertions(+), 4 deletions(-)fs/fs.c +1
-2@@ -579,6 +579,7 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, { struct fstype_info *info = fs_get_info(fs_type); void *buf; + loff_t file_size; int ret; #ifdef CONFIG_LMB@@ -589,10 +590,26 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, } #endif - /* -* We don't actually know how many bytes are being read, since len==0
- * means read the whole file.
Quoted 8 lines
Quoted 2 lines
- */
+ ret =info->size(filename, &file_size);I get an error when running the erofs test cases. The return value isn't as expected
when reading symlink file.
For symlink file,erofs_sizewill return the size of the symlink itself here.
Indeed, this is a problem, in fact I also checked other fses, it looks like we all just return the inode size for the softlink, thus size() call can not be relied on for those cases.
While for the read() calls, every fs will do extra resolving for soft links, thus it doesn't cause problems.
This can still be solved by not calling size() calls at all, and only do the unaligned read handling for the leading block.
Thank you very much for pointing this bug out, would update the patchset for that.
Thanks,
Qu
Quoted 30 lines
Quoted 11 lines
+ if (ret < 0) {
+log_err("** Unable to get file size for %s, %d **\n", + filename, ret);
+ return ret;
+ }
+ if (offset >=file_size) {
+ log_err(
+ "** Invalid offset, offset (%llu) >= file size (%llu)\n", + offset,file_size);
+ return-EINVAL;
+Quoted 8 lines
+ }
+ if (len == 0 || offset + len >file_size) {
+ if (len >file_size)
+ log_info(
+ "** Truncate read length from %llu to %llu, as file size is %llu **\n",
+ len,file_size,file_size);
+ len =file_size- offset;Then, we will get a wrong len in the case of len==0. So I think we need to do something
extra with the symlink file?Thanks,
JiananQuoted 4 lines
+ }
buf =map_sysmem(addr, len);
ret =info->read(filename, buf, offset, len, actread);unmap_sysmem(buf);
RFC PATCH 3/8 fs: btrfs: move the unaligned read code to _fs_read() for btrfs
In reply to Qu WenruoUnlike FUSE or kernel, U-boot filesystem code makes the underly fs code to handle the unaligned read (aka, read range is not aligned to fs block size).
This makes underlying fs code harder to implement, as unlike FUSE/kernel code, now they have to handle unaligned read all by themselves.
This patch will change the behavior, starting from btrfs, by moving the unaligned read code into _fs_read().
The idea is pretty simple, if we have an unaligned read request, we handle it in the following steps:
1. Grab the blocksize of the fs
2. Read the leading unaligned range
We will read the block that @offset is in, and copy the requested part into buf.
The the block we read covers the whole range, we just call it a day.
3. Read the aligned part
4. Read the tailing unaligned range
Pretty much the same as the leading unaligned range, just read the whole block where @offset + @len is, then copy the requested range in the buffer.
This has been tested with a proper randomly populated btrfs file, then tried in sandbox mode with different aligned and unaligned range and compare the output with md5sum.
--- fs/btrfs/btrfs.c | 24 ++++---- fs/btrfs/inode.c | 84 ++------------------------- fs/fs.c | 148 +++++++++++++++++++++++++++++++++++++++++++++-- include/btrfs.h | 1 + 4 files changed, 160 insertions(+), 97 deletions(-)
fs/btrfs/btrfs.c +11 -13
@@ -223,17 +223,27 @@ out:return ret; } +int btrfs_get_blocksize (const char *filename) +{ +struct btrfs_fs_info *fs_info = current_fs_info; + +return fs_info->sectorsize; +} +int btrfs_read (const char *file,void *buf,loff_t offset,loff_t len,loff_t *actread) {struct btrfs_fs_info *fs_info = current_fs_info;struct btrfs_root *root; -loff_t real_size =0 ;u64 ino;u8 type;int ret;ASSERT (fs_info); + +/* Higher layer should always pass correct @len in. */ +ASSERT (len); + ret =btrfs_lookup_path (fs_info->fs_root,BTRFS_FIRST_FREE_OBJECTID , file, &root, &ino, &type,40 );if (ret <0 ) {
@@ -246,18 +256,6 @@ int btrfs_read(const char *file, void *buf, loff_t offset, loff_t len,return -EINVAL ; } -if (!len) { - ret =btrfs_size (file, &real_size); -if (ret <0 ) { -error ("Failed to get inode size: %s" , file); -return ret; - } - len = real_size; - } - -if (len > real_size - offset) - len = real_size - offset; - ret =btrfs_file_read (root, ino, offset, len, buf);if (ret <0 ) {error ("An error occurred while reading file %s" , file);
fs/btrfs/inode.c +6 -78
102 lines not on this page · raw message
fs/fs.c +142 -6
240 lines not on this page · raw message
include/btrfs.h +1 -0
@@ -17,6 +17,7 @@ int btrfs_ls(const char *);int btrfs_exists (const char *);int btrfs_size (const char *,loff_t *);int btrfs_read (const char *,void *,loff_t ,loff_t ,loff_t *); +int btrfs_get_blocksize (const char *);void btrfs_close (void );int btrfs_uuid (char *);void btrfs_list_subvols (void );
Signature · 1 line
2.36.1
RFC PATCH 4/8 fs: ext4: rely on _fs_read() to pass block aligned range into ext4fs_read_file()
In reply to Qu WenruoSince _fs_read() is handling the unaligned read internally, ext4 driver only need to handle block aligned read.
Unfortunately I'm not familiar with ext4 and its driver, thus not confident enough to cleanup all the unaligned read related code.
So here we will have some dead code, and any help to clean them up is appreciated.
--- fs/ext4/ext4fs.c | 11 +++++++++++ fs/fs.c | 2 +- include/ext4fs.h | 1 + 3 files changed, 13 insertions(+), 1 deletion(-)
fs/ext4/ext4fs.c +11 -0
@@ -71,6 +71,10 @@ int ext4fs_read_file(struct ext2fs_node *node, loff_t pos,ext_cache_init (&cache); +/* Higher layer has ensured to pass block aligned range here. */ +assert (IS_ALIGNED (pos, blocksize)); +assert (IS_ALIGNED (len, blocksize)); +/* Adjust len so it we can't read past the end of the file. */ if (len + pos > filesize) len = (filesize - pos);
@@ -183,6 +187,13 @@ int ext4fs_read_file(struct ext2fs_node *node, loff_t pos,return 0 ; } +int ext4fs_get_blocksize (const char *filename) +{ +struct ext_filesystem *fs =get_fs (); + +return fs->blksz; +} +int ext4fs_ls (const char *dirname) {struct ext2fs_node *dirnode =NULL ;
fs/fs.c +1 -1
@@ -236,7 +236,7 @@ static struct fstype_info fstypes[] = { .exists = ext4fs_exists, .size = ext4fs_size, .read = ext4_read_file, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = ext4fs_get_blocksize, #ifdef CONFIG_CMD_EXT4_WRITE .write = ext4_write_file, .ln = ext4fs_create_link,
include/ext4fs.h +1 -0
@@ -146,6 +146,7 @@ int ext4fs_create_link(const char *target, const char *fname);struct ext_filesystem *get_fs (void );int ext4fs_open (const char *filename,loff_t *len);int ext4fs_read (char *buf,loff_t offset,loff_t len,loff_t *actread); +int ext4fs_get_blocksize (const char *filename);int ext4fs_mount (unsigned part_length);void ext4fs_close (void );void ext4fs_reinit_global (void );
Signature · 1 line
2.36.1
RFC PATCH 5/8 fs: fat: rely on higher layer to get block aligned read range
In reply to Qu WenruoJust implement fat_get_blocksize() for fat, so that fat_read_file() always get a block aligned read range.
Unfortunately I'm not experienced enough to cleanup the fat code, thus further cleanup is appreciated.
--- fs/fat/fat.c | 13 +++++++++++++ fs/fs.c | 2 +- include/fat.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)
fs/fat/fat.c +13 -0
@@ -1299,6 +1299,19 @@ int fat_read_file(const char *filename, void *buf, loff_t offset, loff_t len,return ret; } +int fat_get_blocksize (const char *filename) +{ +fsdata fsdata = {0 }; +int ret; + + ret =get_fs_info (&fsdata); +if (ret) +return ret; + +free (fsdata.fatbuf); +return fsdata.sect_size; +} + typedef struct { struct fs_dir_stream parent; struct fs_dirent dirent;
fs/fs.c +1 -1
@@ -207,7 +207,7 @@ static struct fstype_info fstypes[] = { .exists = fat_exists, .size = fat_size, .read = fat_read_file, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = fat_get_blocksize,#if CONFIG_IS_ENABLED (FAT_WRITE ) .write = file_fat_write, .unlink = fat_unlink,
include/fat.h +1 -0
@@ -201,6 +201,7 @@ int file_fat_detectfs(void);int fat_exists (const char *filename);int fat_size (const char *filename,loff_t *size);int file_fat_read (const char *filename,void *buffer,int maxsize); +int fat_get_blocksize (const char *filename);int fat_set_blk_dev (struct blk_desc *rbdd,struct disk_partition *info);int fat_register_device (struct blk_desc *dev_desc,int part_no);
Signature · 1 line
2.36.1
RFC PATCH 6/8 fs: sandboxfs: add sandbox_fs_get_blocksize()
In reply to Qu WenruoThis is to make sandboxfs to report blocksize it supports for _fs_read() to handle unaligned read.
Unlike all other fses, sandboxfs can handle unaligned read/write without any problem since it's calling read()/write(), which doesn't bother the blocksize at all.
This change is mostly to make testing of _fs_read() much easier.
--- arch/sandbox/cpu/os.c | 11 +++++++++++ fs/fs.c | 2 +- fs/sandbox/sandboxfs.c | 14 ++++++++++++++ include/os.h | 8 ++++++++ include/sandboxfs.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-)
arch/sandbox/cpu/os.c +11 -0
@@ -46,6 +46,17 @@ ssize_t os_read(int fd, void *buf, size_t count)return read (fd, buf, count); } +ssize_t os_get_blocksize (int fd) +{ +struct stat stat = {0 }; +int ret; + + ret =fstat (fd, &stat); +if (ret <0 ) +return -errno; +return stat.st_blksize; +} +ssize_t os_write (int fd,const void *buf,size_t count) {return write (fd, buf, count);
fs/fs.c +1 -1
@@ -261,7 +261,7 @@ static struct fstype_info fstypes[] = { .exists = sandbox_fs_exists, .size = sandbox_fs_size, .read = fs_read_sandbox, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = sandbox_fs_get_blocksize, .write = fs_write_sandbox, .uuid = fs_uuid_unsupported, .opendir = fs_opendir_unsupported,
fs/sandbox/sandboxfs.c +14 -0
@@ -55,6 +55,20 @@ int sandbox_fs_read_at(const char *filename, loff_t pos, void *buffer,return ret; } +int sandbox_fs_get_blocksize (const char *filename) +{ +int fd; +int ret; + + fd =os_open (filename,OS_O_RDONLY ); +if (fd <0 ) +return fd; + + ret =os_get_blocksize (fd); +os_close (fd); +return ret; +} +int sandbox_fs_write_at (const char *filename,loff_t pos,void *buffer,loff_t towrite,loff_t *actwrite) {
include/os.h +8 -0
@@ -26,6 +26,14 @@ struct sandbox_state;*/ ssize_t os_read (int fd,void *buf,size_t count); +/** +* Get the optimial blocksize through stat() call. +* +* @fd: File descriptor as returned by os_open() +* Return: >=0 for the blocksize. <0 for error. +*/ +ssize_t os_get_blocksize (int fd); +/** * Access to the OS write() system call *
include/sandboxfs.h +1 -0
@@ -32,6 +32,7 @@ void sandbox_fs_close(void);int sandbox_fs_ls (const char *dirname);int sandbox_fs_exists (const char *filename);int sandbox_fs_size (const char *filename,loff_t *size); +int sandbox_fs_get_blocksize (const char *filename);int fs_read_sandbox (const char *filename,void *buf,loff_t offset,loff_t len,loff_t *actread); int fs_write_sandbox(const char *filename, void *buf, loff_t offset,
Signature · 1 line
2.36.1
Re: [PATCH RFC 6/8] fs: sandboxfs: add sandbox_fs_get_blocksize()
Quoted 12 lines · Qu Wenruo
This is to make sandboxfs to report blocksize it supports for
_fs_read()to handle unaligned read.Unlike all other fses, sandboxfs can handle unaligned read/write without any problem since it's calling read()/write(), which doesn't bother the blocksize at all.
This change is mostly to make testing of
_fs_read()much easier.Cc Simon Glass <[email protected]>Signed-off-by Qu Wenruo <[email protected]>
Quoted 7 lines
--- arch/sandbox/cpu/os.c | 11 +++++++++++ fs/fs.c | 2 +- fs/sandbox/sandboxfs.c | 14 ++++++++++++++ include/os.h | 8 ++++++++ include/sandboxfs.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-)
with a comment as requested below
Quoted 85 lines
arch/sandbox/cpu/os.c +11
-0@@ -46,6 +46,17 @@ ssize_t os_read(int fd, void *buf, size_t count)return read (fd, buf, count); } +ssize_t os_get_blocksize (int fd) +{ +struct stat stat = {0 }; +int ret; + + ret =fstat (fd, &stat); +if (ret <0 ) +return -errno; +return stat.st_blksize; +} +ssize_t os_write (int fd,const void *buf,size_t count) {return write (fd, buf, count);fs/fs.c +1
-1@@ -261,7 +261,7 @@ static struct fstype_info fstypes[] = { .exists = sandbox_fs_exists, .size = sandbox_fs_size, .read = fs_read_sandbox, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = sandbox_fs_get_blocksize, .write = fs_write_sandbox, .uuid = fs_uuid_unsupported, .opendir = fs_opendir_unsupported,fs/sandbox/sandboxfs.c +14
-0@@ -55,6 +55,20 @@ int sandbox_fs_read_at(const char *filename, loff_t pos, void *buffer,return ret; } +int sandbox_fs_get_blocksize (const char *filename) +{ +int fd; +int ret; + + fd =os_open (filename,OS_O_RDONLY ); +if (fd <0 ) +return fd; + + ret =os_get_blocksize (fd); +os_close (fd); +return ret; +} +int sandbox_fs_write_at (const char *filename,loff_t pos,void *buffer,loff_t towrite,loff_t *actwrite) {include/os.h +8
-0@@ -26,6 +26,14 @@ struct sandbox_state;*/ ssize_t os_read (int fd,void *buf,size_t count); +/** +* Get the optimial blocksize through stat() call. +* +* @fd: File descriptor as returned by os_open() +* Return: >=0 for the blocksize. <0 for error. +*/ +ssize_t os_get_blocksize (int fd); +/** * Access to the OS write() system call * include/sandboxfs.h +0
-0
include/sandboxfs.h void sandbox_fs_close(void); · line 35
Quoted 4 lines
include/sandboxfs.h +1
-0@@ -32,6 +32,7 @@ void sandbox_fs_close(void);int sandbox_fs_ls (const char *dirname);int sandbox_fs_exists (const char *filename);int sandbox_fs_size (const char *filename,loff_t *size); +int sandbox_fs_get_blocksize (const char *filename);
Please add a full comment.
Quoted 6 lines
include/sandboxfs.h +0
-1int fs_read_sandbox (const char *filename,void *buf,loff_t offset,loff_t len,loff_t *actread); int fs_write_sandbox(const char *filename, void *buf, loff_t offset, --2.36.1
Regards,
Simon
Re: [PATCH RFC 6/8] fs: sandboxfs: add sandbox_fs_get_blocksize()
Quoted 108 lines · Simon Glass
Quoted 12 lines · Qu Wenruo
This is to make sandboxfs to report blocksize it supports for
_fs_read()to handle unaligned read.Unlike all other fses, sandboxfs can handle unaligned read/write without any problem since it's calling read()/write(), which doesn't bother the blocksize at all.
This change is mostly to make testing of
_fs_read()much easier.Cc Simon Glass <[email protected]>Signed-off-by Qu Wenruo <[email protected]>Quoted 7 lines
--- arch/sandbox/cpu/os.c | 11 +++++++++++ fs/fs.c | 2 +- fs/sandbox/sandboxfs.c | 14 ++++++++++++++ include/os.h | 8 ++++++++ include/sandboxfs.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-)Reviewed-by Simon Glass <[email protected]>with a comment as requested below
Quoted 83 lines
arch/sandbox/cpu/os.c +11
-0@@ -46,6 +46,17 @@ ssize_t os_read(int fd, void *buf, size_t count)return read (fd, buf, count); } +ssize_t os_get_blocksize (int fd) +{ +struct stat stat = {0 }; +int ret; + + ret =fstat (fd, &stat); +if (ret <0 ) +return -errno; +return stat.st_blksize; +} +ssize_t os_write (int fd,const void *buf,size_t count) {return write (fd, buf, count);fs/fs.c +1
-1@@ -261,7 +261,7 @@ static struct fstype_info fstypes[] = { .exists = sandbox_fs_exists, .size = sandbox_fs_size, .read = fs_read_sandbox, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = sandbox_fs_get_blocksize, .write = fs_write_sandbox, .uuid = fs_uuid_unsupported, .opendir = fs_opendir_unsupported,fs/sandbox/sandboxfs.c +14
-0@@ -55,6 +55,20 @@ int sandbox_fs_read_at(const char *filename, loff_t pos, void *buffer,return ret; } +int sandbox_fs_get_blocksize (const char *filename) +{ +int fd; +int ret; + + fd =os_open (filename,OS_O_RDONLY ); +if (fd <0 ) +return fd; + + ret =os_get_blocksize (fd); +os_close (fd); +return ret; +} +int sandbox_fs_write_at (const char *filename,loff_t pos,void *buffer,loff_t towrite,loff_t *actwrite) {include/os.h +8
-0@@ -26,6 +26,14 @@ struct sandbox_state;*/ ssize_t os_read (int fd,void *buf,size_t count); +/** +* Get the optimial blocksize through stat() call. +* +* @fd: File descriptor as returned by os_open() +* Return: >=0 for the blocksize. <0 for error. +*/ +ssize_t os_get_blocksize (int fd); +/** * Access to the OS write() system call * include/sandboxfs.h +0
-1--- a/include/sandboxfs.h
Quoted 8 lines
include/sandboxfs.h void sandbox_fs_close(void); · line 35
Quoted 6 lines
+++
b/include/sandboxfs.hinclude/sandboxfs.h +1
-0@@ -32,6 +32,7 @@ void sandbox_fs_close(void);int sandbox_fs_ls (const char *dirname);int sandbox_fs_exists (const char *filename);int sandbox_fs_size (const char *filename,loff_t *size); +int sandbox_fs_get_blocksize (const char *filename);Please add a full comment.
This is already removed in the formal version:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=654990
As sandbox is just calling host OS read() to handle IO, thus it doesn't need to bother the alignment at all.
Thanks,
Qu
Quoted 10 lines
Quoted 7 lines
include/sandboxfs.h +0
-1int fs_read_sandbox (const char *filename,void *buf,loff_t offset,loff_t len,loff_t *actread); int fs_write_sandbox(const char *filename, void *buf, loff_t offset, --2.36.1
Regards,
Simon
RFC PATCH 7/8 fs: ubifs: rely on higher layer to do unaligned read
In reply to Qu WenruoCurrently ubifs doesn't support unaligned read offset, thanks to the recent _fs_read() work to handle unaligned read, we only need to implement ubifs_get_blocksize() to take advantage of it.
Now ubifs can do unaligned read without any problem.
--- fs/fs.c | 2 +- fs/ubifs/ubifs.c | 13 ++++++++----- include/ubifs_uboot.h | 1 + 3 files changed, 10 insertions(+), 6 deletions(-)
fs/fs.c +1 -1
@@ -302,7 +302,7 @@ static struct fstype_info fstypes[] = { .exists = ubifs_exists, .size = ubifs_size, .read = ubifs_read, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = ubifs_get_blocksize, .write = fs_write_unsupported, .uuid = fs_uuid_unsupported, .opendir = fs_opendir_unsupported,
fs/ubifs/ubifs.c +8 -5
@@ -846,11 +846,9 @@ int ubifs_read(const char *filename, void *buf, loff_t offset, *actread =0 ; -if (offset & (PAGE_SIZE -1 )) { -printf ("ubifs: Error offset must be a multiple of %d\n" , -PAGE_SIZE ); -return -1 ; - } +/* Higher layer should ensure it always pass page aligned range. */ +assert (IS_ALIGNED (offset,PAGE_SIZE )); +assert (IS_ALIGNED (size,PAGE_SIZE )); c->ubi =ubi_open_volume (c->vi.ubi_num, c->vi.vol_id,UBI_READONLY );/* ubifs_findfile will resolve symlinks, so we know that we get
@@ -920,6 +918,11 @@ out:return err; } +int ubifs_get_blocksize (const char *filename) +{ +return PAGE_SIZE ; +} +void ubifs_close (void ) { }
include/ubifs_uboot.h +1 -0
@@ -29,6 +29,7 @@ int ubifs_exists(const char *filename);int ubifs_size (const char *filename,loff_t *size);int ubifs_read (const char *filename,void *buf,loff_t offset,loff_t size,loff_t *actread); +int ubifs_get_blocksize (const char *filename);void ubifs_close (void );#endif /* __UBIFS_UBOOT_H__ */
Signature · 1 line
2.36.1
RFC PATCH 8/8 fs: erofs: add unaligned read range handling
In reply to Qu WenruoI'm not an expert on erofs, but my quick glance didn't expose any special handling on unaligned range, thus I think the U-boot erofs driver doesn't really support unaligned read range.
This patch will add erofs_get_blocksize() so erofs can benefit from the generic unaligned read support.
--- fs/erofs/internal.h | 1 + fs/erofs/super.c | 6 ++++++ fs/fs.c | 2 +- include/erofs.h | 1 + 4 files changed, 9 insertions(+), 1 deletion(-)
fs/erofs/internal.h +1 -0
@@ -83,6 +83,7 @@ struct erofs_sb_info {u16 available_compr_algs;u16 lz4_max_distance;u32 checksum; +u32 blocksize;u16 extra_devices;union {u16 devt_slotoff;/* used for mkfs */
fs/erofs/super.c +6 -0
@@ -99,7 +99,13 @@ int erofs_read_superblock(void) sbi.build_time =le64_to_cpu (dsb->build_time); sbi.build_time_nsec =le32_to_cpu (dsb->build_time_nsec); + sbi.blocksize =1 << blkszbits;memcpy (&sbi.uuid, dsb->uuid,sizeof (dsb->uuid));return erofs_init_devices (&sbi, dsb); } + +int erofs_get_blocksize (const char *filename) +{ +return sbi.blocksize; +}
fs/fs.c +1 -1
@@ -366,7 +366,7 @@ static struct fstype_info fstypes[] = { .readdir = erofs_readdir, .ls = fs_ls_generic, .read = erofs_read, - .get_blocksize = fs_get_blocksize_unsupported, + .get_blocksize = erofs_get_blocksize, .size = erofs_size, .close = erofs_close, .closedir = erofs_closedir,
include/erofs.h +1 -0
@@ -10,6 +10,7 @@ int erofs_probe(struct blk_desc *fs_dev_desc, struct disk_partition *fs_partition);int erofs_read (const char *filename,void *buf,loff_t offset,loff_t len,loff_t *actread); +int erofs_get_blocksize (const char *filename);int erofs_size (const char *filename,loff_t *size);int erofs_exists (const char *filename);void erofs_close (void );
Signature · 1 line
2.36.1
Quoted 63 lines · Qu Wenruo
[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).
But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.
[ADVANTAGE]
This patchset will handle unaligned read range in_fs_read():- Get blocksize of the underlying fs
- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.
If the first block covers the whole range, we just call it aday.
- Read the aligned range if there is any
- Read the tailing block containing the unaligned range And copy the covered range into the destination.
[DISADVANTAGE]
There are mainly two problems:- Extra memory allocation for every
_fs_read()call For the leading and tailing block.- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.[SUPPORTED FSES]
- Btrfs (manually tested*)
- Ext4 (manually tested)
- FAT (manually tested)
- Erofs
- sandboxfs
- ubifs*: Failed to get the test cases run, thus have to go sandbox mode, and attach an image with target fs, load the target file (with unaligned range) and compare the result using md5sum.
For EXT4/FAT, they may need extra cleanup, as their existing unaligned range handling is no longer needed anymore, cleaning them up should free more code lines than the added one.
Just not confident enough to modify them all by myself.
[UNSUPPORTED FSES]
- Squashfs
They don't support non-zero offset, thus it can not handle the tailing block reading.
Need extra help to add block aligned offset support.
Quoted 3 lines
- Semihostfs
It's using hardcoded trap to do system calls, not sure how it would work forstat()call.
There are no alignment requirements for semihosted FSs. So you can pass in an unaligned offset and it will work fine. This is because typically the host will call read() and the host OS will do the aligning.
--Sean
Quoted 30 lines · Qu Wenruo
[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).
But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.
[ADVANTAGE]
This patchset will handle unaligned read range in_fs_read():- Get blocksize of the underlying fs
- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.
If the first block covers the whole range, we just call it aday.
- Read the aligned range if there is any
- Read the tailing block containing the unaligned range And copy the covered range into the destination.
[DISADVANTAGE]
There are mainly two problems:
Quoted 7 lines
- Extra memory allocation for every
_fs_read()call For the leading and tailing block.- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.
This conceptually seems like a good thing. Can you please post some before/after times of reading large images from the supported filesystems?
Signature · 1 line
Tom
Quoted 34 lines · Tom Rini
Quoted 33 lines · Qu Wenruo
[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).
But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.
[ADVANTAGE]
This patchset will handle unaligned read range in_fs_read():- Get blocksize of the underlying fs
- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.
If the first block covers the whole range, we just call it aday.
- Read the aligned range if there is any
- Read the tailing block containing the unaligned range And copy the covered range into the destination.
[DISADVANTAGE]
There are mainly two problems:- Extra memory allocation for every
_fs_read()call For the leading and tailing block.
Quoted 9 lines
Quoted 4 lines
- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.This conceptually seems like a good thing. Can you please post some before/after times of reading large images from the supported filesystems?
One thing to mention is, this change doesn't really bother large file read.
As the patchset is splitting a large read into 3 parts:
1) Leading block
2) Aligned blocks, aka the main part of a large file
3) Tailing block
Most time should still be spent on part 2), not much different than the old code. Part 1) and Part 3) are at most 2 blocks (aka, 2 * 4KiB for most modern large enough fses).
So I doubt it would make any difference for large file read.
Furthermore, as pointed out by Huang Jianan, currently the patchset can not handle read on soft link correctly, thus I'd update the series to do the split into even less parts:
1) Leading block
For the unaligned initial block
2) Aligned blocks until the end
The tailing block should still starts at a block aligned position, thus most filesystems is already handling them correctly. (Just a min(end, blockend) is enough for most cases already).
Anyway, I'll try to craft some benchmarking for file reads using sandbox. But please don't expect much (or any) difference in that case.
Thanks,
Qu
Quoted 69 lines · Qu Wenruo
Quoted 34 lines · Tom Rini
Quoted 33 lines · Qu Wenruo
[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).
But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.
[ADVANTAGE]
This patchset will handle unaligned read range in_fs_read():- Get blocksize of the underlying fs
- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.
If the first block covers the whole range, we just call it aday.
- Read the aligned range if there is any
- Read the tailing block containing the unaligned range And copy the covered range into the destination.
[DISADVANTAGE]
There are mainly two problems:- Extra memory allocation for every
_fs_read()call For the leading and tailing block.Quoted 9 lines
Quoted 4 lines
- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.This conceptually seems like a good thing. Can you please post some before/after times of reading large images from the supported filesystems?
One thing to mention is, this change doesn't really bother large file read.
As the patchset is splitting a large read into 3 parts:
1) Leading block
2) Aligned blocks, aka the main part of a large file
3) Tailing blockMost time should still be spent on part 2), not much different than the old code. Part 1) and Part 3) are at most 2 blocks (aka, 2 * 4KiB for most modern large enough fses).
So I doubt it would make any difference for large file read.
Furthermore, as pointed out by Huang Jianan, currently the patchset can not handle read on soft link correctly, thus I'd update the series to do the split into even less parts:
1) Leading block
For the unaligned initial block
Quoted 7 lines
2) Aligned blocks until the end
The tailing block should still starts at a block aligned position, thus most filesystems is already handling them correctly. (Just amin(end, blockend)is enough for most cases already).Anyway, I'll try to craft some benchmarking for file reads using sandbox. But please don't expect much (or any) difference in that case.
The rework sounds good. And doing it without any real impact to performance either way is good.
Signature · 1 line
Tom