Skip to contentSkip to conversation

u-boot: fs: add generic unaligned read handling

PATCH In review Qu Wenruo 6 participants 18 messages Jun 28, 2022 – Jun 30, 2022

Subsystems (6)
lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover u-boot: fs: add generic unaligned read handling

[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.

Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).

But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.

[ADVANTAGE]
This patchset will handle unaligned read range in _fs_read():

- Get blocksize of the underlying fs

- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.

If the first block covers the whole range, we just call it aday.

- Read the aligned range if there is any

- Read the tailing block containing the unaligned range And copy the covered range into the destination.

[DISADVANTAGE]
There are mainly two problems:

- Extra memory allocation for every _fs_read() call For the leading and tailing block.

- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.

[SUPPORTED FSES]

- Btrfs (manually tested*)
- Ext4 (manually tested)
- FAT (manually tested)
- Erofs
- sandboxfs
- ubifs

*: Failed to get the test cases run, thus have to go sandbox mode, and attach an image with target fs, load the target file (with unaligned range) and compare the result using md5sum.

For EXT4/FAT, they may need extra cleanup, as their existing unaligned range handling is no longer needed anymore, cleaning them up should free more code lines than the added one.

Just not confident enough to modify them all by myself.

[UNSUPPORTED FSES]
- Squashfs
They don't support non-zero offset, thus it can not handle the tailing block reading.
Need extra help to add block aligned offset support.

- Semihostfs
It's using hardcoded trap to do system calls, not sure how it would work for stat() call.

Extra testing/feedback is always appreciated.

Qu Wenruo (8):
fs: fat: unexport file_fat_read_at()
fs: always get the file size in _fs_read()
fs: btrfs: move the unaligned read code to _fs_read() for btrfs
fs: ext4: rely on _fs_read() to pass block aligned range into

    ext4fs_read_file()

fs: fat: rely on higher layer to get block aligned read range
fs: sandboxfs: add sandbox_fs_get_blocksize()
fs: ubifs: rely on higher layer to do unaligned read
fs: erofs: add unaligned read range handling

 arch/sandbox/cpu/os.c  |  11 +++
 fs/btrfs/btrfs.c       |  24 +++---
 fs/btrfs/inode.c       |  84 ++------------------
 fs/erofs/internal.h    |   1 +
 fs/erofs/super.c       |   6 ++
 fs/ext4/ext4fs.c       |  11 +++
 fs/fat/fat.c           |  17 ++++-
 fs/fs.c                | 169 +++++++++++++++++++++++++++++++++++++++--
 fs/sandbox/sandboxfs.c |  14 ++++
 fs/ubifs/ubifs.c       |  13 ++--
 include/btrfs.h        |   1 +
 include/erofs.h        |   1 +
 include/ext4fs.h       |   1 +
 include/fat.h          |   3 +-
 include/os.h           |   8 ++
 include/sandboxfs.h    |   1 +
 include/ubifs_uboot.h  |   1 +
 17 files changed, 258 insertions(+), 108 deletions(-)
Signature · 1 line
2.36.1
author
#1 lore raw

RFC PATCH 1/8 fs: fat: unexport file_fat_read_at()

That function is only utilized inside fat driver, unexport it.

---
 fs/fat/fat.c  | 4 ++--
 include/fat.h | 2 --
 2 files changed, 2 insertions(+), 4 deletions(-)
fs/fat/fat.c +2 -2
index df9ea2c028fc..dcceccbcee0a 100644
@@ -1243,8 +1243,8 @@ out_free_itr:
 	return ret;
 }
 
-int file_fat_read_at(const char *filename, loff_t pos, void *buffer,
-		     loff_t maxsize, loff_t *actread)
+static int file_fat_read_at(const char *filename, loff_t pos, void *buffer,
+			    loff_t maxsize, loff_t *actread)
 {
 	fsdata fsdata;
 	fat_itr *itr;
include/fat.h +0 -2
index bd8e450b33a3..a9756fb4cd1b 100644
@@ -200,8 +200,6 @@ static inline u32 sect_to_clust(fsdata *fsdata, int sect)
 int file_fat_detectfs(void);
 int fat_exists(const char *filename);
 int fat_size(const char *filename, loff_t *size);
-int file_fat_read_at(const char *filename, loff_t pos, void *buffer,
-		     loff_t maxsize, loff_t *actread);
 int file_fat_read(const char *filename, void *buffer, int maxsize);
 int fat_set_blk_dev(struct blk_desc *rbdd, struct disk_partition *info);
 int fat_register_device(struct blk_desc *dev_desc, int part_no);
Signature · 1 line
2.36.1
Reviewed-by on v1 1/8
#2 lore raw

Re: [PATCH RFC 1/8] fs: fat: unexport file_fat_read_at()

Quoted 8 lines · Qu Wenruo

That function is only utilized inside fat driver, unexport it.

---
 fs/fat/fat.c  | 4 ++--
 include/fat.h | 2 --
 2 files changed, 2 insertions(+), 4 deletions(-)
Quoted 32 lines
fs/fat/fat.c +2 -2
index df9ea2c028fc..dcceccbcee0a 100644
@@ -1243,8 +1243,8 @@ out_free_itr:
        return ret;
 }
 
-int file_fat_read_at(const char *filename, loff_t pos, void *buffer,
-                    loff_t maxsize, loff_t *actread)
+static int file_fat_read_at(const char *filename, loff_t pos, void *buffer,
+                           loff_t maxsize, loff_t *actread)
 {
        fsdata fsdata;
        fat_itr *itr;
include/fat.h +0 -3
index bd8e450b33a3..a9756fb4cd1b 100644
@@ -200,8 +200,6 @@ static inline u32 sect_to_clust(fsdata *fsdata, int sect)
 int file_fat_detectfs(void);
 int fat_exists(const char *filename);
 int fat_size(const char *filename, loff_t *size);
-int file_fat_read_at(const char *filename, loff_t pos, void *buffer,
-                    loff_t maxsize, loff_t *actread);
 int file_fat_read(const char *filename, void *buffer, int maxsize);
 int fat_set_blk_dev(struct blk_desc *rbdd, struct disk_partition *info);
 int fat_register_device(struct blk_desc *dev_desc, int part_no);
--

2.36.1

author
#3 lore raw

RFC PATCH 2/8 fs: always get the file size in _fs_read()

In reply to Qu Wenruo

For _fs_read(), @len == 0 means we read the whole file. And we just pass @len == 0 to make each filesystem to handle it.

In fact we have info->size() call to properly get the filesize.

So we can not only call info->size() to grab the file_size for len == 0 case, but also detect invalid @len (e.g. @len > file_size) in advance or truncate @len.

This behavior also allows us to handle unaligned better in the incoming patches.

---
 fs/fs.c | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)
fs/fs.c +21 -4
index 6de1a3eb6d5d..d992cdd6d650 100644
@@ -579,6 +579,7 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len,
 {
 	struct fstype_info *info = fs_get_info(fs_type);
 	void *buf;
+	loff_t file_size;
 	int ret;
 
 #ifdef CONFIG_LMB
@@ -589,10 +590,26 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len,
 	}
 #endif
 
-	/*
-	 * We don't actually know how many bytes are being read, since len==0
-	 * means read the whole file.
-	 */
+	ret = info->size(filename, &file_size);
+	if (ret < 0) {
+		log_err("** Unable to get file size for %s, %d **\n",
+				filename, ret);
+		return ret;
+	}
+	if (offset >= file_size) {
+		log_err(
+		"** Invalid offset, offset (%llu) >= file size (%llu)\n",
+			offset, file_size);
+		return -EINVAL;
+
+	}
+	if (len == 0 || offset + len > file_size) {
+		if (len > file_size)
+			log_info(
+	"** Truncate read length from %llu to %llu, as file size is %llu **\n",
+				 len, file_size, file_size);
+		len = file_size - offset;
+	}
 	buf = map_sysmem(addr, len);
 	ret = info->read(filename, buf, offset, len, actread);
 	unmap_sysmem(buf);
Signature · 1 line
2.36.1
#4 lore raw

Re: [PATCH RFC 2/8] fs: always get the file size in _fs_read()

Hi, wenruo,

Quoted 30 lines · Qu Wenruo

For _fs_read(), @len == 0 means we read the whole file. And we just pass @len == 0 to make each filesystem to handle it.

In fact we have info->size() call to properly get the filesize.

So we can not only call info->size() to grab the file_size for len == 0 case, but also detect invalid @len (e.g. @len > file_size) in advance or truncate @len.

This behavior also allows us to handle unaligned better in the incoming patches.

---
  fs/fs.c | 25 +++++++++++++++++++++----
  1 file changed, 21 insertions(+), 4 deletions(-)
fs/fs.c +1 -0
index 6de1a3eb6d5d..d992cdd6d650 100644
@@ -579,6 +579,7 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len,
  {
  	struct fstype_info *info = fs_get_info(fs_type);
  	void *buf;
+	loff_t file_size;
  	int ret;
 
  #ifdef CONFIG_LMB

fs/fs.c static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, · line 593

Quoted 8 lines
fs/fs.c +1 -4
@@ -589,10 +590,26 @@ static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len,
  	}
  #endif
 
-	/*
-	 * We don't actually know how many bytes are being read, since len==0
-	 * means read the whole file.
-	 */
+	ret = info->size(filename, &file_size);

I get an error when running the erofs test cases. The return value isn't as expected when reading symlink file.
For symlink file, erofs_size will return the size of the symlink itself here.

Quoted 8 lines
fs/fs.c +8 -0
+	if (ret < 0) {
+		log_err("** Unable to get file size for %s, %d **\n",
+				filename, ret);
+		return ret;
+	}
+	if (offset >= file_size) {
+		log_err(
+		"** Invalid offset, offset (%llu) >= file size (%llu)\n",

fs/fs.c static int _fs_read(const char *filename, ulong addr, loff_t offset, loff_t len, · line 611

Quoted 10 lines
fs/fs.c +10 -0
+			offset, file_size);
+		return -EINVAL;
+
+	}
+	if (len == 0 || offset + len > file_size) {
+		if (len > file_size)
+			log_info(
+	"** Truncate read length from %llu to %llu, as file size is %llu **\n",
+				 len, file_size, file_size);
+		len = file_size - offset;

Then, we will get a wrong len in the case of len==0. So I think we need to do something extra with the symlink file?

Thanks,
Jianan

Quoted 4 lines
fs/fs.c +1 -0
+	}
  	buf = map_sysmem(addr, len);
  	ret = info->read(filename, buf, offset, len, actread);
  	unmap_sysmem(buf);
#5 lore raw

Re: [PATCH RFC 2/8] fs: always get the file size in _fs_read()

Quoted 40 lines · Huang Jianan

Hi, wenruo,

Quoted 37 lines · Qu Wenruo

For _fs_read(), @len == 0 means we read the whole file. And we just pass @len == 0 to make each filesystem to handle it.

In fact we have info->size() call to properly get the filesize.

So we can not only call info->size() to grab the file_size for len == 0 case, but also detect invalid @len (e.g. @len > file_size) in advance or truncate @len.

This behavior also allows us to handle unaligned better in the incoming patches.

---
  fs/fs.c | 25 +++++++++++++++++++++----

  1 file changed, 21 insertions(+), 4 deletions(-)
fs/fs.c +1 -2
index 6de1a3eb6d5d..d992cdd6d650 100644
@@ -579,6 +579,7 @@ static int _fs_read(const char *filename, ulong
 addr, loff_t offset, loff_t len,
   {
       struct fstype_info *info = fs_get_info(fs_type);
       void *buf;
+    loff_t file_size;
       int ret;
   #ifdef CONFIG_LMB
@@ -589,10 +590,26 @@ static int _fs_read(const char *filename, ulong
 addr, loff_t offset, loff_t len,
       }
   #endif
-    /*
-     * We don't actually know how many bytes are being read, since

len==0
-     * means read the whole file.

Quoted 8 lines
Quoted 2 lines

-     */
+    ret = info->size(filename, &file_size);

I get an error when running the erofs test cases. The return value isn't as expected
when reading symlink file.
For symlink file, erofs_size will return the size of the symlink itself here.

Indeed, this is a problem, in fact I also checked other fses, it looks like we all just return the inode size for the softlink, thus size() call can not be relied on for those cases.

While for the read() calls, every fs will do extra resolving for soft links, thus it doesn't cause problems.

This can still be solved by not calling size() calls at all, and only do the unaligned read handling for the leading block.

Thank you very much for pointing this bug out, would update the patchset for that.

Thanks,
Qu

Quoted 30 lines
Quoted 11 lines

+    if (ret < 0) {
+        log_err("** Unable to get file size for %s, %d **\n", +                filename, ret);
+        return ret;
+    }
+    if (offset >= file_size) {
+        log_err(
+        "** Invalid offset, offset (%llu) >= file size (%llu)\n", +            offset, file_size);
+        return -EINVAL;
+

Quoted 8 lines

+    }
+    if (len == 0 || offset + len > file_size) {
+        if (len > file_size)
+            log_info(
+    "** Truncate read length from %llu to %llu, as file size is %llu **\n",
+                 len, file_size, file_size);
+        len = file_size - offset;

Then, we will get a wrong len in the case of len==0. So I think we need to do something
extra with the symlink file?

Thanks,
Jianan

Quoted 4 lines

+    }
      buf = map_sysmem(addr, len);
      ret = info->read(filename, buf, offset, len, actread); unmap_sysmem(buf);

author
#6 lore raw

RFC PATCH 3/8 fs: btrfs: move the unaligned read code to _fs_read() for btrfs

In reply to Qu Wenruo

Unlike FUSE or kernel, U-boot filesystem code makes the underly fs code to handle the unaligned read (aka, read range is not aligned to fs block size).

This makes underlying fs code harder to implement, as unlike FUSE/kernel code, now they have to handle unaligned read all by themselves.

This patch will change the behavior, starting from btrfs, by moving the unaligned read code into _fs_read().

The idea is pretty simple, if we have an unaligned read request, we handle it in the following steps:

1. Grab the blocksize of the fs

2. Read the leading unaligned range
We will read the block that @offset is in, and copy the requested part into buf.

The the block we read covers the whole range, we just call it a day.

3. Read the aligned part

4. Read the tailing unaligned range
Pretty much the same as the leading unaligned range, just read the whole block where @offset + @len is, then copy the requested range in the buffer.

This has been tested with a proper randomly populated btrfs file, then tried in sandbox mode with different aligned and unaligned range and compare the output with md5sum.

---
 fs/btrfs/btrfs.c |  24 ++++----
 fs/btrfs/inode.c |  84 ++-------------------------
 fs/fs.c          | 148 +++++++++++++++++++++++++++++++++++++++++++++--
 include/btrfs.h  |   1 +
 4 files changed, 160 insertions(+), 97 deletions(-)
fs/btrfs/btrfs.c +11 -13
index 741c6e20f533..f9a67468d508 100644
@@ -223,17 +223,27 @@ out:
 	return ret;
 }
 
+int btrfs_get_blocksize(const char *filename)
+{
+	struct btrfs_fs_info *fs_info = current_fs_info;
+
+	return fs_info->sectorsize;
+}
+
 int btrfs_read(const char *file, void *buf, loff_t offset, loff_t len,
 	       loff_t *actread)
 {
 	struct btrfs_fs_info *fs_info = current_fs_info;
 	struct btrfs_root *root;
-	loff_t real_size = 0;
 	u64 ino;
 	u8 type;
 	int ret;
 
 	ASSERT(fs_info);
+
+	/* Higher layer should always pass correct @len in. */
+	ASSERT(len);
+
 	ret = btrfs_lookup_path(fs_info->fs_root, BTRFS_FIRST_FREE_OBJECTID,
 				file, &root, &ino, &type, 40);
 	if (ret < 0) {
@@ -246,18 +256,6 @@ int btrfs_read(const char *file, void *buf, loff_t offset, loff_t len,
 		return -EINVAL;
 	}
 
-	if (!len) {
-		ret = btrfs_size(file, &real_size);
-		if (ret < 0) {
-			error("Failed to get inode size: %s", file);
-			return ret;
-		}
-		len = real_size;
-	}
-
-	if (len > real_size - offset)
-		len = real_size - offset;
-
 	ret = btrfs_file_read(root, ino, offset, len, buf);
 	if (ret < 0) {
 		error("An error occurred while reading file %s", file);
fs/btrfs/inode.c +6 -78

102 lines not on this page · raw message

fs/fs.c +142 -6

240 lines not on this page · raw message

include/btrfs.h +1 -0
index a7605e158970..bba71ec02893 100644
@@ -17,6 +17,7 @@ int btrfs_ls(const char *);
 int btrfs_exists(const char *);
 int btrfs_size(const char *, loff_t *);
 int btrfs_read(const char *, void *, loff_t, loff_t, loff_t *);
+int btrfs_get_blocksize(const char *);
 void btrfs_close(void);
 int btrfs_uuid(char *);
 void btrfs_list_subvols(void);
Signature · 1 line
2.36.1
author
#7 lore raw

RFC PATCH 4/8 fs: ext4: rely on _fs_read() to pass block aligned range into ext4fs_read_file()

In reply to Qu Wenruo

Since _fs_read() is handling the unaligned read internally, ext4 driver only need to handle block aligned read.

Unfortunately I'm not familiar with ext4 and its driver, thus not confident enough to cleanup all the unaligned read related code.

So here we will have some dead code, and any help to clean them up is appreciated.

---
 fs/ext4/ext4fs.c | 11 +++++++++++
 fs/fs.c          |  2 +-
 include/ext4fs.h |  1 +
 3 files changed, 13 insertions(+), 1 deletion(-)
fs/ext4/ext4fs.c +11 -0
index 4c89152ce4ad..be2680994d8b 100644
@@ -71,6 +71,10 @@ int ext4fs_read_file(struct ext2fs_node *node, loff_t pos,
 
 	ext_cache_init(&cache);
 
+	/* Higher layer has ensured to pass block aligned range here. */
+	assert(IS_ALIGNED(pos, blocksize));
+	assert(IS_ALIGNED(len, blocksize));
+
 	/* Adjust len so it we can't read past the end of the file. */
 	if (len + pos > filesize)
 		len = (filesize - pos);
@@ -183,6 +187,13 @@ int ext4fs_read_file(struct ext2fs_node *node, loff_t pos,
 	return 0;
 }
 
+int ext4fs_get_blocksize(const char *filename)
+{
+	struct ext_filesystem *fs = get_fs();
+
+	return fs->blksz;
+}
+
 int ext4fs_ls(const char *dirname)
 {
 	struct ext2fs_node *dirnode = NULL;
fs/fs.c +1 -1
index 30696ac6c1a3..e69a0968bb6d 100644
@@ -236,7 +236,7 @@ static struct fstype_info fstypes[] = {
 		.exists = ext4fs_exists,
 		.size = ext4fs_size,
 		.read = ext4_read_file,
-		.get_blocksize = fs_get_blocksize_unsupported,
+		.get_blocksize = ext4fs_get_blocksize,
 #ifdef CONFIG_CMD_EXT4_WRITE
 		.write = ext4_write_file,
 		.ln = ext4fs_create_link,
include/ext4fs.h +1 -0
index cb5d9cc0a5c0..cc40cfedd954 100644
@@ -146,6 +146,7 @@ int ext4fs_create_link(const char *target, const char *fname);
 struct ext_filesystem *get_fs(void);
 int ext4fs_open(const char *filename, loff_t *len);
 int ext4fs_read(char *buf, loff_t offset, loff_t len, loff_t *actread);
+int ext4fs_get_blocksize(const char *filename);
 int ext4fs_mount(unsigned part_length);
 void ext4fs_close(void);
 void ext4fs_reinit_global(void);
Signature · 1 line
2.36.1
author
#8 lore raw

RFC PATCH 5/8 fs: fat: rely on higher layer to get block aligned read range

In reply to Qu Wenruo

Just implement fat_get_blocksize() for fat, so that fat_read_file() always get a block aligned read range.

Unfortunately I'm not experienced enough to cleanup the fat code, thus further cleanup is appreciated.

---
 fs/fat/fat.c  | 13 +++++++++++++
 fs/fs.c       |  2 +-
 include/fat.h |  1 +
 3 files changed, 15 insertions(+), 1 deletion(-)
fs/fat/fat.c +13 -0
index dcceccbcee0a..e13035e8e6d1 100644
@@ -1299,6 +1299,19 @@ int fat_read_file(const char *filename, void *buf, loff_t offset, loff_t len,
 	return ret;
 }
 
+int fat_get_blocksize(const char *filename)
+{
+	fsdata fsdata = {0};
+	int ret;
+
+	ret = get_fs_info(&fsdata);
+	if (ret)
+		return ret;
+
+	free(fsdata.fatbuf);
+	return fsdata.sect_size;
+}
+
 typedef struct {
 	struct fs_dir_stream parent;
 	struct fs_dirent dirent;
fs/fs.c +1 -1
index e69a0968bb6d..7e4ead9b790b 100644
@@ -207,7 +207,7 @@ static struct fstype_info fstypes[] = {
 		.exists = fat_exists,
 		.size = fat_size,
 		.read = fat_read_file,
-		.get_blocksize = fs_get_blocksize_unsupported,
+		.get_blocksize = fat_get_blocksize,
 #if CONFIG_IS_ENABLED(FAT_WRITE)
 		.write = file_fat_write,
 		.unlink = fat_unlink,
include/fat.h +1 -0
index a9756fb4cd1b..c03a2bebecef 100644
@@ -201,6 +201,7 @@ int file_fat_detectfs(void);
 int fat_exists(const char *filename);
 int fat_size(const char *filename, loff_t *size);
 int file_fat_read(const char *filename, void *buffer, int maxsize);
+int fat_get_blocksize(const char *filename);
 int fat_set_blk_dev(struct blk_desc *rbdd, struct disk_partition *info);
 int fat_register_device(struct blk_desc *dev_desc, int part_no);
 
Signature · 1 line
2.36.1
author
#9 lore raw

RFC PATCH 6/8 fs: sandboxfs: add sandbox_fs_get_blocksize()

In reply to Qu Wenruo

This is to make sandboxfs to report blocksize it supports for _fs_read() to handle unaligned read.

Unlike all other fses, sandboxfs can handle unaligned read/write without any problem since it's calling read()/write(), which doesn't bother the blocksize at all.

This change is mostly to make testing of _fs_read() much easier.

---
 arch/sandbox/cpu/os.c  | 11 +++++++++++
 fs/fs.c                |  2 +-
 fs/sandbox/sandboxfs.c | 14 ++++++++++++++
 include/os.h           |  8 ++++++++
 include/sandboxfs.h    |  1 +
 5 files changed, 35 insertions(+), 1 deletion(-)
arch/sandbox/cpu/os.c +11 -0
index 5ea54179176c..6c29f29bdd9b 100644
@@ -46,6 +46,17 @@ ssize_t os_read(int fd, void *buf, size_t count)
 	return read(fd, buf, count);
 }
 
+ssize_t os_get_blocksize(int fd)
+{
+	struct stat stat = {0};
+	int ret;
+
+	ret = fstat(fd, &stat);
+	if (ret < 0)
+		return -errno;
+	return stat.st_blksize;
+}
+
 ssize_t os_write(int fd, const void *buf, size_t count)
 {
 	return write(fd, buf, count);
fs/fs.c +1 -1
index 7e4ead9b790b..337d5711c28c 100644
@@ -261,7 +261,7 @@ static struct fstype_info fstypes[] = {
 		.exists = sandbox_fs_exists,
 		.size = sandbox_fs_size,
 		.read = fs_read_sandbox,
-		.get_blocksize = fs_get_blocksize_unsupported,
+		.get_blocksize = sandbox_fs_get_blocksize,
 		.write = fs_write_sandbox,
 		.uuid = fs_uuid_unsupported,
 		.opendir = fs_opendir_unsupported,
fs/sandbox/sandboxfs.c +14 -0
index 4ae41d5b4db1..130fee088621 100644
@@ -55,6 +55,20 @@ int sandbox_fs_read_at(const char *filename, loff_t pos, void *buffer,
 	return ret;
 }
 
+int sandbox_fs_get_blocksize(const char *filename)
+{
+	int fd;
+	int ret;
+
+	fd = os_open(filename, OS_O_RDONLY);
+	if (fd < 0)
+		return fd;
+
+	ret = os_get_blocksize(fd);
+	os_close(fd);
+	return ret;
+}
+
 int sandbox_fs_write_at(const char *filename, loff_t pos, void *buffer,
 			loff_t towrite, loff_t *actwrite)
 {
include/os.h +8 -0
index 10e198cf503e..a864d9ca39b2 100644
@@ -26,6 +26,14 @@ struct sandbox_state;
  */
 ssize_t os_read(int fd, void *buf, size_t count);
 
+/**
+ * Get the optimial blocksize through stat() call.
+ *
+ * @fd:		File descriptor as returned by os_open()
+ * Return:	>=0 for the blocksize. <0 for error.
+ */
+ssize_t os_get_blocksize(int fd);
+
 /**
  * Access to the OS write() system call
  *
include/sandboxfs.h +1 -0
index 783dd5c88a73..6937068f7b82 100644
@@ -32,6 +32,7 @@ void sandbox_fs_close(void);
 int sandbox_fs_ls(const char *dirname);
 int sandbox_fs_exists(const char *filename);
 int sandbox_fs_size(const char *filename, loff_t *size);
+int sandbox_fs_get_blocksize(const char *filename);
 int fs_read_sandbox(const char *filename, void *buf, loff_t offset, loff_t len,
 		    loff_t *actread);
 int fs_write_sandbox(const char *filename, void *buf, loff_t offset,
Signature · 1 line
2.36.1
Reviewed-by on v1 6/8
#10 lore raw

Re: [PATCH RFC 6/8] fs: sandboxfs: add sandbox_fs_get_blocksize()

Quoted 12 lines · Qu Wenruo

This is to make sandboxfs to report blocksize it supports for _fs_read() to handle unaligned read.

Unlike all other fses, sandboxfs can handle unaligned read/write without any problem since it's calling read()/write(), which doesn't bother the blocksize at all.

This change is mostly to make testing of _fs_read() much easier.

Quoted 7 lines
---
 arch/sandbox/cpu/os.c  | 11 +++++++++++
 fs/fs.c                |  2 +-
 fs/sandbox/sandboxfs.c | 14 ++++++++++++++
 include/os.h           |  8 ++++++++
 include/sandboxfs.h    |  1 +
 5 files changed, 35 insertions(+), 1 deletion(-)

with a comment as requested below

Quoted 85 lines
arch/sandbox/cpu/os.c +11 -0
index 5ea54179176c..6c29f29bdd9b 100644
@@ -46,6 +46,17 @@ ssize_t os_read(int fd, void *buf, size_t count)
        return read(fd, buf, count);
 }
 
+ssize_t os_get_blocksize(int fd)
+{
+       struct stat stat = {0};
+       int ret;
+
+       ret = fstat(fd, &stat);
+       if (ret < 0)
+               return -errno;
+       return stat.st_blksize;
+}
+
 ssize_t os_write(int fd, const void *buf, size_t count)
 {
        return write(fd, buf, count);
fs/fs.c +1 -1
index 7e4ead9b790b..337d5711c28c 100644
@@ -261,7 +261,7 @@ static struct fstype_info fstypes[] = {
                .exists = sandbox_fs_exists,
                .size = sandbox_fs_size,
                .read = fs_read_sandbox,
-               .get_blocksize = fs_get_blocksize_unsupported,
+               .get_blocksize = sandbox_fs_get_blocksize,
                .write = fs_write_sandbox,
                .uuid = fs_uuid_unsupported,
                .opendir = fs_opendir_unsupported,
fs/sandbox/sandboxfs.c +14 -0
index 4ae41d5b4db1..130fee088621 100644
@@ -55,6 +55,20 @@ int sandbox_fs_read_at(const char *filename, loff_t pos, void *buffer,
        return ret;
 }
 
+int sandbox_fs_get_blocksize(const char *filename)
+{
+       int fd;
+       int ret;
+
+       fd = os_open(filename, OS_O_RDONLY);
+       if (fd < 0)
+               return fd;
+
+       ret = os_get_blocksize(fd);
+       os_close(fd);
+       return ret;
+}
+
 int sandbox_fs_write_at(const char *filename, loff_t pos, void *buffer,
                        loff_t towrite, loff_t *actwrite)
 {
include/os.h +8 -0
index 10e198cf503e..a864d9ca39b2 100644
@@ -26,6 +26,14 @@ struct sandbox_state;
  */
 ssize_t os_read(int fd, void *buf, size_t count);
 
+/**
+ * Get the optimial blocksize through stat() call.
+ *
+ * @fd:                File descriptor as returned by os_open()
+ * Return:     >=0 for the blocksize. <0 for error.
+ */
+ssize_t os_get_blocksize(int fd);
+
 /**
  * Access to the OS write() system call
  *
include/sandboxfs.h +0 -0
index 783dd5c88a73..6937068f7b82 100644

include/sandboxfs.h void sandbox_fs_close(void); · line 35

Quoted 4 lines
include/sandboxfs.h +1 -0
@@ -32,6 +32,7 @@ void sandbox_fs_close(void);
 int sandbox_fs_ls(const char *dirname);
 int sandbox_fs_exists(const char *filename);
 int sandbox_fs_size(const char *filename, loff_t *size);
+int sandbox_fs_get_blocksize(const char *filename);

Please add a full comment.

Quoted 6 lines
include/sandboxfs.h +0 -1
 int fs_read_sandbox(const char *filename, void *buf, loff_t offset, loff_t len,
                    loff_t *actread);
 int fs_write_sandbox(const char *filename, void *buf, loff_t offset,
--

2.36.1

Regards,
Simon

author
#11 lore raw

Re: [PATCH RFC 6/8] fs: sandboxfs: add sandbox_fs_get_blocksize()

Quoted 108 lines · Simon Glass
Quoted 12 lines · Qu Wenruo

This is to make sandboxfs to report blocksize it supports for _fs_read() to handle unaligned read.

Unlike all other fses, sandboxfs can handle unaligned read/write without any problem since it's calling read()/write(), which doesn't bother the blocksize at all.

This change is mostly to make testing of _fs_read() much easier.

Quoted 7 lines
---
  arch/sandbox/cpu/os.c  | 11 +++++++++++
  fs/fs.c                |  2 +-
  fs/sandbox/sandboxfs.c | 14 ++++++++++++++
  include/os.h           |  8 ++++++++
  include/sandboxfs.h    |  1 +
  5 files changed, 35 insertions(+), 1 deletion(-)

with a comment as requested below

Quoted 83 lines
arch/sandbox/cpu/os.c +11 -0
index 5ea54179176c..6c29f29bdd9b 100644
@@ -46,6 +46,17 @@ ssize_t os_read(int fd, void *buf, size_t count)
         return read(fd, buf, count);
  }
 
+ssize_t os_get_blocksize(int fd)
+{
+       struct stat stat = {0};
+       int ret;
+
+       ret = fstat(fd, &stat);
+       if (ret < 0)
+               return -errno;
+       return stat.st_blksize;
+}
+
  ssize_t os_write(int fd, const void *buf, size_t count)
  {
         return write(fd, buf, count);
fs/fs.c +1 -1
index 7e4ead9b790b..337d5711c28c 100644
@@ -261,7 +261,7 @@ static struct fstype_info fstypes[] = {
                 .exists = sandbox_fs_exists,
                 .size = sandbox_fs_size,
                 .read = fs_read_sandbox,
-               .get_blocksize = fs_get_blocksize_unsupported,
+               .get_blocksize = sandbox_fs_get_blocksize,
                 .write = fs_write_sandbox,
                 .uuid = fs_uuid_unsupported,
                 .opendir = fs_opendir_unsupported,
fs/sandbox/sandboxfs.c +14 -0
index 4ae41d5b4db1..130fee088621 100644
@@ -55,6 +55,20 @@ int sandbox_fs_read_at(const char *filename, loff_t pos, void *buffer,
         return ret;
  }
 
+int sandbox_fs_get_blocksize(const char *filename)
+{
+       int fd;
+       int ret;
+
+       fd = os_open(filename, OS_O_RDONLY);
+       if (fd < 0)
+               return fd;
+
+       ret = os_get_blocksize(fd);
+       os_close(fd);
+       return ret;
+}
+
  int sandbox_fs_write_at(const char *filename, loff_t pos, void *buffer,
                         loff_t towrite, loff_t *actwrite)
  {
include/os.h +8 -0
index 10e198cf503e..a864d9ca39b2 100644
@@ -26,6 +26,14 @@ struct sandbox_state;
   */
  ssize_t os_read(int fd, void *buf, size_t count);
 
+/**
+ * Get the optimial blocksize through stat() call.
+ *
+ * @fd:                File descriptor as returned by os_open()
+ * Return:     >=0 for the blocksize. <0 for error.
+ */
+ssize_t os_get_blocksize(int fd);
+
  /**
   * Access to the OS write() system call
   *
include/sandboxfs.h +0 -1
index 783dd5c88a73..6937068f7b82 100644
--- a/include/sandboxfs.h
Quoted 8 lines

include/sandboxfs.h void sandbox_fs_close(void); · line 35

Quoted 6 lines

+++ b/include/sandboxfs.h

include/sandboxfs.h +1 -0
@@ -32,6 +32,7 @@ void sandbox_fs_close(void);
  int sandbox_fs_ls(const char *dirname);
  int sandbox_fs_exists(const char *filename);
  int sandbox_fs_size(const char *filename, loff_t *size);
+int sandbox_fs_get_blocksize(const char *filename);

Please add a full comment.

This is already removed in the formal version:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=654990

As sandbox is just calling host OS read() to handle IO, thus it doesn't need to bother the alignment at all.

Thanks,
Qu

Quoted 10 lines
Quoted 7 lines
include/sandboxfs.h +0 -1
  int fs_read_sandbox(const char *filename, void *buf, loff_t offset, loff_t len,
                     loff_t *actread);
  int fs_write_sandbox(const char *filename, void *buf, loff_t offset,
--

2.36.1

Regards,
Simon

author
#12 lore raw

RFC PATCH 7/8 fs: ubifs: rely on higher layer to do unaligned read

In reply to Qu Wenruo

Currently ubifs doesn't support unaligned read offset, thanks to the recent _fs_read() work to handle unaligned read, we only need to implement ubifs_get_blocksize() to take advantage of it.

Now ubifs can do unaligned read without any problem.

---
 fs/fs.c               |  2 +-
 fs/ubifs/ubifs.c      | 13 ++++++++-----
 include/ubifs_uboot.h |  1 +
 3 files changed, 10 insertions(+), 6 deletions(-)
fs/fs.c +1 -1
index 337d5711c28c..41943e3a95db 100644
@@ -302,7 +302,7 @@ static struct fstype_info fstypes[] = {
 		.exists = ubifs_exists,
 		.size = ubifs_size,
 		.read = ubifs_read,
-		.get_blocksize = fs_get_blocksize_unsupported,
+		.get_blocksize = ubifs_get_blocksize,
 		.write = fs_write_unsupported,
 		.uuid = fs_uuid_unsupported,
 		.opendir = fs_opendir_unsupported,
fs/ubifs/ubifs.c +8 -5
index d3026e310168..a8ab556dd376 100644
@@ -846,11 +846,9 @@ int ubifs_read(const char *filename, void *buf, loff_t offset,
 
 	*actread = 0;
 
-	if (offset & (PAGE_SIZE - 1)) {
-		printf("ubifs: Error offset must be a multiple of %d\n",
-		       PAGE_SIZE);
-		return -1;
-	}
+	/* Higher layer should ensure it always pass page aligned range. */
+	assert(IS_ALIGNED(offset, PAGE_SIZE));
+	assert(IS_ALIGNED(size, PAGE_SIZE));
 
 	c->ubi = ubi_open_volume(c->vi.ubi_num, c->vi.vol_id, UBI_READONLY);
 	/* ubifs_findfile will resolve symlinks, so we know that we get
@@ -920,6 +918,11 @@ out:
 	return err;
 }
 
+int ubifs_get_blocksize(const char *filename)
+{
+	return PAGE_SIZE;
+}
+
 void ubifs_close(void)
 {
 }
include/ubifs_uboot.h +1 -0
index b025779d59ff..bcd21715314a 100644
@@ -29,6 +29,7 @@ int ubifs_exists(const char *filename);
 int ubifs_size(const char *filename, loff_t *size);
 int ubifs_read(const char *filename, void *buf, loff_t offset,
 	       loff_t size, loff_t *actread);
+int ubifs_get_blocksize(const char *filename);
 void ubifs_close(void);
 
 #endif /* __UBIFS_UBOOT_H__ */
Signature · 1 line
2.36.1
author
#13 lore raw

RFC PATCH 8/8 fs: erofs: add unaligned read range handling

In reply to Qu Wenruo

I'm not an expert on erofs, but my quick glance didn't expose any special handling on unaligned range, thus I think the U-boot erofs driver doesn't really support unaligned read range.

This patch will add erofs_get_blocksize() so erofs can benefit from the generic unaligned read support.

---
 fs/erofs/internal.h | 1 +
 fs/erofs/super.c    | 6 ++++++
 fs/fs.c             | 2 +-
 include/erofs.h     | 1 +
 4 files changed, 9 insertions(+), 1 deletion(-)
fs/erofs/internal.h +1 -0
index 4af7c91560cc..d368a6481bf1 100644
@@ -83,6 +83,7 @@ struct erofs_sb_info {
 	u16 available_compr_algs;
 	u16 lz4_max_distance;
 	u32 checksum;
+	u32 blocksize;
 	u16 extra_devices;
 	union {
 		u16 devt_slotoff;		/* used for mkfs */
fs/erofs/super.c +6 -0
index 4cca322b9ead..df01d2e719a7 100644
@@ -99,7 +99,13 @@ int erofs_read_superblock(void)
 
 	sbi.build_time = le64_to_cpu(dsb->build_time);
 	sbi.build_time_nsec = le32_to_cpu(dsb->build_time_nsec);
+	sbi.blocksize = 1 << blkszbits;
 
 	memcpy(&sbi.uuid, dsb->uuid, sizeof(dsb->uuid));
 	return erofs_init_devices(&sbi, dsb);
 }
+
+int erofs_get_blocksize(const char *filename)
+{
+	return sbi.blocksize;
+}
fs/fs.c +1 -1
index 41943e3a95db..0ef5fbdda5d6 100644
@@ -366,7 +366,7 @@ static struct fstype_info fstypes[] = {
 		.readdir = erofs_readdir,
 		.ls = fs_ls_generic,
 		.read = erofs_read,
-		.get_blocksize = fs_get_blocksize_unsupported,
+		.get_blocksize = erofs_get_blocksize,
 		.size = erofs_size,
 		.close = erofs_close,
 		.closedir = erofs_closedir,
include/erofs.h +1 -0
index 1fbe82bf72cb..18bd6807c538 100644
@@ -10,6 +10,7 @@ int erofs_probe(struct blk_desc *fs_dev_desc,
 		struct disk_partition *fs_partition);
 int erofs_read(const char *filename, void *buf, loff_t offset,
 	       loff_t len, loff_t *actread);
+int erofs_get_blocksize(const char *filename);
 int erofs_size(const char *filename, loff_t *size);
 int erofs_exists(const char *filename);
 void erofs_close(void);
Signature · 1 line
2.36.1
#14 lore raw
In reply to Qu Wenruo
Quoted 63 lines · Qu Wenruo

[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.

Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).

But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.

[ADVANTAGE]
This patchset will handle unaligned read range in _fs_read():

- Get blocksize of the underlying fs

- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.

If the first block covers the whole range, we just call it aday.

- Read the aligned range if there is any

- Read the tailing block containing the unaligned range And copy the covered range into the destination.

[DISADVANTAGE]
There are mainly two problems:

- Extra memory allocation for every _fs_read() call For the leading and tailing block.

- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.

[SUPPORTED FSES]

- Btrfs (manually tested*)
- Ext4 (manually tested)
- FAT (manually tested)
- Erofs
- sandboxfs
- ubifs

*: Failed to get the test cases run, thus have to go sandbox mode, and attach an image with target fs, load the target file (with unaligned range) and compare the result using md5sum.

For EXT4/FAT, they may need extra cleanup, as their existing unaligned range handling is no longer needed anymore, cleaning them up should free more code lines than the added one.

Just not confident enough to modify them all by myself.

[UNSUPPORTED FSES]
- Squashfs
They don't support non-zero offset, thus it can not handle the tailing block reading.
Need extra help to add block aligned offset support.

Quoted 3 lines

- Semihostfs
It's using hardcoded trap to do system calls, not sure how it would work for stat() call.

There are no alignment requirements for semihosted FSs. So you can pass in an unaligned offset and it will work fine. This is because typically the host will call read() and the host OS will do the aligning.

--Sean

#15 lore raw
In reply to Qu Wenruo
Quoted 30 lines · Qu Wenruo

[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.

Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).

But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.

[ADVANTAGE]
This patchset will handle unaligned read range in _fs_read():

- Get blocksize of the underlying fs

- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.

If the first block covers the whole range, we just call it aday.

- Read the aligned range if there is any

- Read the tailing block containing the unaligned range And copy the covered range into the destination.

[DISADVANTAGE]
There are mainly two problems:

Quoted 7 lines

- Extra memory allocation for every _fs_read() call For the leading and tailing block.

- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.

This conceptually seems like a good thing. Can you please post some before/after times of reading large images from the supported filesystems?

Signature · 1 line
Tom
#16 lore raw
Quoted 34 lines · Tom Rini
Quoted 33 lines · Qu Wenruo

[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.

Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).

But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.

[ADVANTAGE]
This patchset will handle unaligned read range in _fs_read():

- Get blocksize of the underlying fs

- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.

If the first block covers the whole range, we just call it aday.

- Read the aligned range if there is any

- Read the tailing block containing the unaligned range And copy the covered range into the destination.

[DISADVANTAGE]
There are mainly two problems:

- Extra memory allocation for every _fs_read() call For the leading and tailing block.

Quoted 9 lines
Quoted 4 lines

- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.

This conceptually seems like a good thing. Can you please post some before/after times of reading large images from the supported filesystems?

One thing to mention is, this change doesn't really bother large file read.

As the patchset is splitting a large read into 3 parts:

1) Leading block
2) Aligned blocks, aka the main part of a large file
3) Tailing block

Most time should still be spent on part 2), not much different than the old code. Part 1) and Part 3) are at most 2 blocks (aka, 2 * 4KiB for most modern large enough fses).

So I doubt it would make any difference for large file read.

Furthermore, as pointed out by Huang Jianan, currently the patchset can not handle read on soft link correctly, thus I'd update the series to do the split into even less parts:

1) Leading block
For the unaligned initial block

2) Aligned blocks until the end
The tailing block should still starts at a block aligned position, thus most filesystems is already handling them correctly. (Just a min(end, blockend) is enough for most cases already).

Anyway, I'll try to craft some benchmarking for file reads using sandbox. But please don't expect much (or any) difference in that case.

Thanks,
Qu

#17 lore raw
Quoted 69 lines · Qu Wenruo
Quoted 34 lines · Tom Rini
Quoted 33 lines · Qu Wenruo

[BACKGROUND]
Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses.

Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size).

But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case.

[ADVANTAGE]
This patchset will handle unaligned read range in _fs_read():

- Get blocksize of the underlying fs

- Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer.

If the first block covers the whole range, we just call it aday.

- Read the aligned range if there is any

- Read the tailing block containing the unaligned range And copy the covered range into the destination.

[DISADVANTAGE]
There are mainly two problems:

- Extra memory allocation for every _fs_read() call For the leading and tailing block.

Quoted 9 lines
Quoted 4 lines

- Extra path resolving
All those supported fs will have to do extra path resolving up to 2 times (one for the leading block, one for the tailing block). This may slow down the read.

This conceptually seems like a good thing. Can you please post some before/after times of reading large images from the supported filesystems?

One thing to mention is, this change doesn't really bother large file read.

As the patchset is splitting a large read into 3 parts:

1) Leading block
2) Aligned blocks, aka the main part of a large file
3) Tailing block

Most time should still be spent on part 2), not much different than the old code. Part 1) and Part 3) are at most 2 blocks (aka, 2 * 4KiB for most modern large enough fses).

So I doubt it would make any difference for large file read.

Furthermore, as pointed out by Huang Jianan, currently the patchset can not handle read on soft link correctly, thus I'd update the series to do the split into even less parts:

1) Leading block
For the unaligned initial block

Quoted 7 lines

2) Aligned blocks until the end
The tailing block should still starts at a block aligned position, thus most filesystems is already handling them correctly. (Just a min(end, blockend) is enough for most cases already).

Anyway, I'll try to craft some benchmarking for file reads using sandbox. But please don't expect much (or any) difference in that case.

The rework sounds good. And doing it without any real impact to performance either way is good.

Signature · 1 line
Tom