squash4: fix handling of fragments and sparse files
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 squash4: fix handling of fragments and sparse files
1. Do not assume block list and fragment are mutually exclusive. Squash can pack file tail as fragment (unless -no-fragments is specified); so check read offset and read either from block list or from fragments as appropriate.
2. Support sparse files with zero blocks.
3. Fix fragment read - frag.offset is absolute fragment position, not offset relative to ino.chunk.
Reported and tested by Carlo Caione <[email protected]>
---
@Vladimir: we need regression tests for both of these cases. We could real small files only by accident - block list was zero, so it appeared to work.
@Carlo, please test. I'm surprised it worked for you even without fragments as your file is sparse and grub immediately choked on the first zero block.
grub-core/fs/squash4.c | 55 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 19 deletions(-)
grub-core/fs/squash4.c +36 -19
@@ -823,7 +823,12 @@ direct_read (struct grub_squash_data *data, curread = data->blksz - boff;if (curread > len) curread = len; -if (!(ino->block_sizes[i] +if (!ino->block_sizes[i]) + { +/* Sparse block */ +grub_memset (buf,'\0' , curread); + } +else if (!(ino->block_sizes[i] &grub_cpu_to_le32_compile_time (SQUASH_BLOCK_UNCOMPRESSED ))) {char *block;
@@ -873,36 +878,57 @@ direct_read (struct grub_squash_data *data,static grub_ssize_t -grub_squash_read_data (struct grub_squash_data *data, -struct grub_squash_cache_inode *ino, -grub_off_t off,char *buf,grub_size_t len) +grub_squash_read (grub_file_t file,char *buf,grub_size_t len) { +struct grub_squash_data *data = file->data; +struct grub_squash_cache_inode *ino = &data->ino; +grub_off_t off = file->offset;grub_err_t err;grub_uint64_t a =0 , b;grub_uint32_t fragment =0 ;int compressed =0 ;struct grub_squash_frag_desc frag; +grub_off_t blk_len; +grub_uint64_t mask =grub_le_to_cpu32 (data->sb.block_size) -1 ; +grub_size_t orig_len = len;switch (ino->ino.type) {case grub_cpu_to_le16_compile_time (SQUASH_TYPE_LONG_REGULAR ): - a =grub_le_to_cpu64 (ino->ino.long_file.chunk); fragment =grub_le_to_cpu32 (ino->ino.long_file.fragment);break ;case grub_cpu_to_le16_compile_time (SQUASH_TYPE_REGULAR ): - a =grub_le_to_cpu32 (ino->ino.file.chunk); fragment =grub_le_to_cpu32 (ino->ino.file.fragment);break ; } -if (fragment ==0xffffffff ) -return direct_read (data, ino, off, buf, len); +/* Squash may pack file tail as fragment. So read initial part directly and +get tail from fragments */ + blk_len = fragment ==0xffffffff ? file->size : file->size & ~mask; +if (off < blk_len) + { +grub_size_t read_len = blk_len - off; +grub_ssize_t res; + +if (read_len > len) + read_len = len; + res =direct_read (data, ino, off, buf, read_len); +if ((grub_size_t ) res != read_len) +return -1 ;/* FIXME: is short read possible here? */ + len -= read_len; +if (!len) +return read_len; + buf += read_len; + off =0 ; + } +else + off -= blk_len; err =read_chunk (data, &frag,sizeof (frag), data->fragments,sizeof (frag) * fragment);if (err)return -1 ; - a +=grub_le_to_cpu64 (frag.offset); + a =grub_le_to_cpu64 (frag.offset); compressed = !(frag.size &grub_cpu_to_le32_compile_time (SQUASH_BLOCK_UNCOMPRESSED ));if (ino->ino.type ==grub_cpu_to_le16_compile_time (SQUASH_TYPE_LONG_REGULAR )) b =grub_le_to_cpu32 (ino->ino.long_file.offset) + off;
@@ -943,16 +969,7 @@ grub_squash_read_data (struct grub_squash_data *data,if (err)return -1 ; } -return len; -} - -static grub_ssize_t -grub_squash_read (grub_file_t file,char *buf,grub_size_t len) -{ -struct grub_squash_data *data = file->data; - -return grub_squash_read_data (data, &data->ino, - file->offset, buf, len); +return orig_len; } static grub_err_t
Signature · 1 line
tg: (2fb8cd2..) u/squash-tail-fragment (depends on: master)
Quoted 12 lines · Andrei Borzenkov
1. Do not assume block list and fragment are mutually exclusive. Squash can pack file tail as fragment (unless -no-fragments is specified); so check read offset and read either from block list or from fragments as appropriate.
2. Support sparse files with zero blocks.
3. Fix fragment read - frag.offset is absolute fragment position, not offset relative to ino.chunk.
Reported and tested by Carlo Caione <[email protected]>
Quoted 7 lines
---
@Vladimir: we need regression tests for both of these cases. We could real small files only by accident - block list was zero, so it appeared to work.
@Carlo, please test. I'm surprised it worked for you even without fragments as your file is sparse and grub immediately choked on the first zero block.
Just tested. And it works fine for me.
Thanks,
Signature · 1 line
Carlo Caione | +39.340.80.30.096 | Endless
Quoted 10 lines · On Sat, Feb 18, 2017, 10:17 Andrei Borzenkov <[email protected]> wrote:
1. Do not assume block list and fragment are mutually exclusive. Squash can pack file tail as fragment (unless -no-fragments is specified); so check read offset and read either from block list or from fragments as appropriate.
2. Support sparse files with zero blocks.
3. Fix fragment read - frag.offset is absolute fragment position, not offset relative to ino.chunk.
Go ahead.
Quoted 9 lines
Reported and tested by Carlo Caione <[email protected]>
---
@Vladimir: we need regression tests for both of these cases. We could real small
files only by accident - block list was zero, so it appeared to work.
How do you create those files reliably? Feel free to add any files to grub-fs-tester. Feel free to commit without tests, we can add tests later.
Quoted 122 lines
@Carlo, please test. I'm surprised it worked for you even without fragments as
your file is sparse and grub immediately choked on the first zero block.grub-core/fs/squash4.c | 55 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 19 deletions(-)grub-core/fs/squash4.c +14
-8@@ -823,7 +823,12 @@ direct_read (struct grub_squash_data *data, curread = data->blksz - boff;if (curread > len) curread = len; -if (!(ino->block_sizes[i] +if (!ino->block_sizes[i]) + { +/* Sparse block */ +grub_memset (buf,'\0' , curread); + } +else if (!(ino->block_sizes[i] &grub_cpu_to_le32_compile_time (SQUASH_BLOCK_UNCOMPRESSED ))) {char *block;@@ -873,36 +878,57 @@ direct_read (struct grub_squash_data *data,static grub_ssize_t -grub_squash_read_data (struct grub_squash_data *data, -struct grub_squash_cache_inode *ino, -grub_off_t off,char *buf,grub_size_t len) +grub_squash_read (grub_file_t file,char *buf,grub_size_t len) { +struct grub_squash_data *data = file->data; +struct grub_squash_cache_inode *ino = &data->ino; +grub_off_t off = file->offset;grub_err_t err;grub_uint64_t a =0 , b;grub_uint32_t fragment =0 ;int compressed =0 ;struct grub_squash_frag_desc frag; +grub_off_t blk_len; +grub_uint64_t mask =grub_le_to_cpu32 (data->sb.block_size) -1 ; +grub_size_t orig_len = len;switch (ino->ino.type) {case grub_cpu_to_le16_compile_time (SQUASH_TYPE_LONG_REGULAR ): - a =grub_le_to_cpu64 (ino->ino.long_file.chunk); fragment =grub_le_to_cpu32 (ino->ino.long_file.fragment);break ;case grub_cpu_to_le16_compile_time (SQUASH_TYPE_REGULAR ): - a =grub_le_to_cpu32 (ino->ino.file.chunk); fragment =grub_le_to_cpu32 (ino->ino.file.fragment);break ; } -if (fragment ==0xffffffff ) -return direct_read (data, ino, off, buf, len); +/* Squash may pack file tail as fragment. So read initial part directly and
grub-core/fs/squash4.c +21
-1+get tail from fragments */ + blk_len = fragment ==0xffffffff ? file->size : file->size & ~mask; +if (off < blk_len) + { +grub_size_t read_len = blk_len - off; +grub_ssize_t res; + +if (read_len > len) + read_len = len; + res =direct_read (data, ino, off, buf, read_len); +if ((grub_size_t ) res != read_len) +return -1 ;/* FIXME: is short read possible here? */ + len -= read_len; +if (!len) +return read_len; + buf += read_len; + off =0 ; + } +else + off -= blk_len; err =read_chunk (data, &frag,sizeof (frag), data->fragments,sizeof (frag) * fragment);if (err)return -1 ; - a +=grub_le_to_cpu64 (frag.offset); + a =grub_le_to_cpu64 (frag.offset); compressed = !(frag.size & grub_cpu_to_le32_compile_time(SQUASH_BLOCK_UNCOMPRESSED)); if (ino->ino.type == grub_cpu_to_le16_compile_time (SQUASH_TYPE_LONG_REGULAR)) b = grub_le_to_cpu32 (ino->ino.long_file.offset) + off;grub-core/fs/squash4.c +1
-11@@ -943,16 +969,7 @@ grub_squash_read_data (struct grub_squash_data *data,if (err)return -1 ; } -return len; -} - -static grub_ssize_t -grub_squash_read (grub_file_t file,char *buf,grub_size_t len) -{ -struct grub_squash_data *data = file->data; - -return grub_squash_read_data (data, &data->ino, - file->offset, buf, len); +return orig_len; } static grub_err_t --tg: (2fb8cd2..) u/squash-tail-fragment (depends on: master)
_______________________________________________
Bug-grub mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/bug-grub
23.02.2017 16:52, Vladimir 'phcoder' Serbinenko пишет:
Quoted 7 lines · On Sat, Feb 18, 2017, 10:17 Andrei Borzenkov <[email protected]> wrote:
Quoted 6 lines
1. Do not assume block list and fragment are mutually exclusive. Squash can pack file tail as fragment (unless -no-fragments is specified); so check read offset and read either from block list or from fragments as appropriate.
Quoted 7 lines
Quoted 5 lines
2. Support sparse files with zero blocks.
3. Fix fragment read - frag.offset is absolute fragment position, not offset relative to ino.chunk.
Go ahead.
Done with cosmetic changes (renamed one variable to better match surrounding code and removed now superfluous assignment).
Quoted 12 lines
Quoted 9 lines
Reported and tested by Carlo Caione <[email protected]>
---
@Vladimir: we need regression tests for both of these cases. We could real small
files only by accident - block list was zero, so it appeared to work.How do you create those files reliably? Feel free to add any files to grub-fs-tester. Feel free to commit without tests, we can add tests later.
I would say, we need to generally create less "nice" test files
1. Generate test files that are not exact multiple of filesystem block. This would cover squash tail packing and may uncover similar issues in other drivers as well.
2. Generate sparse files by seeking beyond end of file. We already had similar problem with indirect ext* blocks that was not caught by tests. We should produce holes both for direct and indirect blocks (we may pick common offset or make it fs-specific if necessary).
3. We need to produce really small files also to test inlining. Again we can pick common size or make it fs-specific.
Assuming we generate files as described, for squash4 just call it with -always-use-fragments to force tail packing. -nopad may be interesting as well to stress our driver even more.
24.02.2017 19:19, Andrei Borzenkov пишет:
Quoted 15 lines
Quoted 3 lines
How do you create those files reliably? Feel free to add any files to grub-fs-tester. Feel free to commit without tests, we can add tests later.
I would say, we need to generally create less "nice" test files
1. Generate test files that are not exact multiple of filesystem block. This would cover squash tail packing and may uncover similar issues in other drivers as well.
2. Generate sparse files by seeking beyond end of file. We already had similar problem with indirect ext* blocks that was not caught by tests. We should produce holes both for direct and indirect blocks (we may pick common offset or make it fs-specific if necessary).
Quoted 8 lines
3. We need to produce really small files also to test inlining. Again we can pick common size or make it fs-specific.
Assuming we generate files as described, for squash4 just call it with -always-use-fragments to force tail packing. -nopad may be interesting as well to stress our driver even more.
Attached patch implements 1 + -always-use-fragments. It reliably causes squash4 test to fail without my patch. I can change BLOCKCNT for squash4 only, but I think it makes sense to do globally.
sparse files need some tweaking in garbage-gen, will look into it.
From: Andrei Borzenkov <[email protected]>
Subject: [PATCH] grub-fs-tester: improve squash4 tests
1. Make sure files are not multiple of block size. This will ensure tail packing for squash4 and may also trigger more codes paths in other filesystems.
2. Call mksquashfs with -always-use-fragments to force tail packing.
--- tests/util/grub-fs-tester.in | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
tests/util/grub-fs-tester.in +4 -2
@@ -913,6 +913,8 @@ for ((LOGSECSIZE=MINLOGSECSIZE;LOGSECSIZE<=MAXLOGSECSIZE;LOGSECSIZE=LOGSECSIZE + *) BLOCKCNT=5242880;; esac + # Make sure file is not exact multiple of block size + : $((BLOCKCNT--)) case x"$fs" in x"ntfscomp") setfattr -h -v 0x00000800 -n system.ntfs_attrib_be "$MNTPOINTRW/$OSDIR";;
@@ -998,8 +1000,8 @@ for ((LOGSECSIZE=MINLOGSECSIZE;LOGSECSIZE<=MAXLOGSECSIZE;LOGSECSIZE=LOGSECSIZE + x"romfs") genromfs -V "$FSLABEL" -f "${FSIMAGES[0]}" -d "$MASTER" ;; xsquash4_*) - echo mksquashfs "$MASTER" "${FSIMAGES[0]}" -comp "${fs/squash4_/}" -b $BLKSIZE - mksquashfs "$MASTER" "${FSIMAGES[0]}" -comp "${fs/squash4_/}" -b $BLKSIZE ;; + echo mksquashfs "$MASTER" "${FSIMAGES[0]}" -always-use-fragments -nopad -comp "${fs/squash4_/}" -b $BLKSIZE + mksquashfs "$MASTER" "${FSIMAGES[0]}" -always-use-fragments -nopad -comp "${fs/squash4_/}" -b $BLKSIZE ;; x"bfs") sleep 1 fusermount -u "$MNTPOINTRW"
Signature · 1 line
tg: (892dfbe..) u/squash4-tests (depends on: master)
sqaush4 tests (was: Re: [PATCH] squash4: fix handling of fragments and sparse files)
25.02.2017 18:23, Andrei Borzenkov пишет:
Quoted 25 lines
24.02.2017 19:19, Andrei Borzenkov пишет:
Quoted 15 lines
Quoted 3 lines
How do you create those files reliably? Feel free to add any files to grub-fs-tester. Feel free to commit without tests, we can add tests later.
I would say, we need to generally create less "nice" test files
1. Generate test files that are not exact multiple of filesystem block. This would cover squash tail packing and may uncover similar issues in other drivers as well.
2. Generate sparse files by seeking beyond end of file. We already had similar problem with indirect ext* blocks that was not caught by tests. We should produce holes both for direct and indirect blocks (we may pick common offset or make it fs-specific if necessary).
Quoted 8 lines
3. We need to produce really small files also to test inlining. Again we can pick common size or make it fs-specific.
Assuming we generate files as described, for squash4 just call it with -always-use-fragments to force tail packing. -nopad may be interesting as well to stress our driver even more.
Quoted 6 lines
Attached patch implements 1 + -always-use-fragments. It reliably causes squash4 test to fail without my patch. I can change BLOCKCNT for squash4 only, but I think it makes sense to do globally.
sparse files need some tweaking in garbage-gen, will look into it.
Sorry, wrong patch attached.
From: Andrei Borzenkov <[email protected]>
Subject: [PATCH] grub-fs-tester: improve squash4 tests
1. Make sure files are not multiple of block size. This will ensure tail packing for squash4 and may also trigger more codes paths in other filesystems.
2. Call mksquashfs with -always-use-fragments to force tail packing.
--- tests/util/grub-fs-tester.in | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tests/util/grub-fs-tester.in +5 -2
@@ -913,6 +913,9 @@ for ((LOGSECSIZE=MINLOGSECSIZE;LOGSECSIZE<=MAXLOGSECSIZE;LOGSECSIZE=LOGSECSIZE + *) BLOCKCNT=5242880;; esac + # Make sure file is not exact multiple of block size. This helps to force + # tail packing in case of squash4. + : $((BLOCKCNT--)) case x"$fs" in x"ntfscomp") setfattr -h -v 0x00000800 -n system.ntfs_attrib_be "$MNTPOINTRW/$OSDIR";;
@@ -998,8 +1001,8 @@ for ((LOGSECSIZE=MINLOGSECSIZE;LOGSECSIZE<=MAXLOGSECSIZE;LOGSECSIZE=LOGSECSIZE + x"romfs") genromfs -V "$FSLABEL" -f "${FSIMAGES[0]}" -d "$MASTER" ;; xsquash4_*) - echo mksquashfs "$MASTER" "${FSIMAGES[0]}" -comp "${fs/squash4_/}" -b $BLKSIZE - mksquashfs "$MASTER" "${FSIMAGES[0]}" -comp "${fs/squash4_/}" -b $BLKSIZE ;; + echo mksquashfs "$MASTER" "${FSIMAGES[0]}" -always-use-fragments -comp "${fs/squash4_/}" -b $BLKSIZE + mksquashfs "$MASTER" "${FSIMAGES[0]}" -always-use-fragments -comp "${fs/squash4_/}" -b $BLKSIZE ;; x"bfs") sleep 1 fusermount -u "$MNTPOINTRW"
Signature · 1 line
tg: (892dfbe..) u/squash4-tests (depends on: master)