vvfat: Fix write bugs for large files and add iotests
Subsystems (2)
v3 cover vvfat: Fix write bugs for large files and add iotests
These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.
Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash on abort() when the file is modified and a new cluster was added.
Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.
v3:
Added test for creating new files in vvfat.
v2:
Added iotests for vvfat driver along with a simple fat16 module to run the tests.
v1:
https://patchew.org/QEMU/[email protected]/
Fix the issue of writing to the middle of the file in vvfat
Amjad Alsharafi (6):
vvfat: Fix bug in writing to middle of file
vvfat: Fix usage of info.file.offset
vvfat: Fix reading files with non-continuous clusters
iotests: Add vvfat tests
iotests: Filter out vvfat fmt from failing tests
iotests: Add create_file test for vvfat driver
.gitlab-ci.d/buildtest.yml | 1 + block/vvfat.c | 32 +- tests/qemu-iotests/001 | 1 + tests/qemu-iotests/002 | 1 + tests/qemu-iotests/003 | 1 + tests/qemu-iotests/005 | 1 + tests/qemu-iotests/008 | 1 + tests/qemu-iotests/009 | 1 + tests/qemu-iotests/010 | 1 + tests/qemu-iotests/011 | 1 + tests/qemu-iotests/012 | 1 + tests/qemu-iotests/021 | 1 + tests/qemu-iotests/032 | 1 + tests/qemu-iotests/033 | 1 + tests/qemu-iotests/052 | 1 + tests/qemu-iotests/094 | 1 + tests/qemu-iotests/120 | 2 +- tests/qemu-iotests/140 | 1 + tests/qemu-iotests/145 | 1 + tests/qemu-iotests/157 | 1 + tests/qemu-iotests/159 | 2 +- tests/qemu-iotests/170 | 2 +- tests/qemu-iotests/192 | 1 + tests/qemu-iotests/197 | 2 +- tests/qemu-iotests/208 | 2 +- tests/qemu-iotests/215 | 2 +- tests/qemu-iotests/236 | 2 +- tests/qemu-iotests/251 | 1 + tests/qemu-iotests/307 | 2 +- tests/qemu-iotests/308 | 2 +- tests/qemu-iotests/check | 2 +- tests/qemu-iotests/fat16.py | 619 ++++++++++++++++++ tests/qemu-iotests/meson.build | 3 +- .../tests/export-incoming-iothread | 2 +- tests/qemu-iotests/tests/fuse-allow-other | 1 + .../tests/mirror-ready-cancel-error | 2 +- tests/qemu-iotests/tests/regression-vhdx-log | 1 + tests/qemu-iotests/tests/vvfat | 419 ++++++++++++ tests/qemu-iotests/tests/vvfat.out | 5 + 39 files changed, 1098 insertions(+), 26 deletions(-) create mode 100644 tests/qemu-iotests/fat16.py create mode 100755 tests/qemu-iotests/tests/vvfat create mode 100755 tests/qemu-iotests/tests/vvfat.out
Signature · 1 line
2.45.0
PATCH v3 1/6 vvfat: Fix bug in writing to middle of file
Before this commit, the behavior when calling commit_one_file for example with offset=0x2000 (second cluster), what will happen is that we won't fetch the next cluster from the fat, and instead use the first cluster for the read operation.
This is due to off-by-one error here, where i=0x2000 !< offset=0x2000, thus not fetching the next cluster.
--- block/vvfat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/vvfat.c +1 -1
@@ -2525,7 +2525,7 @@ commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset)return -1 ; } -for (i = s->cluster_size; i < offset; i += s->cluster_size) +for (i = s->cluster_size; i <= offset; i += s->cluster_size) c =modified_fat_get (s, c); fd =qemu_open_old (mapping->path,O_RDWR |O_CREAT |O_BINARY ,0666 );
Signature · 1 line
2.45.0
Re: [PATCH v3 1/6] vvfat: Fix bug in writing to middle of file
Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:
Quoted 9 lines
Before this commit, the behavior when calling
commit_one_filefor example withoffset=0x2000(second cluster), what will happen is that we won't fetch the next cluster from the fat, and instead use the first cluster for the read operation.This is due to off-by-one error here, where
i=0x2000 !< offset=0x2000, thus not fetching the next cluster.Signed-off-by Amjad Alsharafi <[email protected]>
block/vvfat.c commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset) · line 2529
Quoted 11 lines
block/vvfat.c +1
-1@@ -2525,7 +2525,7 @@ commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset)return -1 ; } -for (i = s->cluster_size; i < offset; i += s->cluster_size) +for (i = s->cluster_size; i <= offset; i += s->cluster_size) c =modified_fat_get (s, c);
While your change results in the correct behaviour, I think I would prefer the code to be changed like this so that at the start of each loop iteration, 'i' always refers to the offset that matches 'c':
for (i = 0; i < offset; i += s->cluster_size) {
c = modified_fat_get(s, c);
}I'm also adding braces here to make the code conform with the QEMU coding style. You can use scripts/checkpatch.pl to make sure that all code you add has the correct style. Much of the vvfat code predates the coding style, so you'll often have to change style when you touch a line. (Which is good, because it slowly fixes the existing mess.)
You can keep my Reviewed/Tested-by if you change this.
Kevin
PATCH v3 2/6 vvfat: Fix usage of `info.file.offset`
In reply to Amjad AlsharafiThe field is marked as "the offset in the file (in clusters)", but it was being used like this cluster_size*(nums)+mapping->info.file.offset, which is incorrect.
Additionally, removed the abort when first_mapping_index does not match, as this matches the case when adding new clusters for files, and its inevitable that we reach this condition when doing that if the clusters are not after one another, so there is no reason to abort here, execution continues and the new clusters are written to disk correctly.
--- block/vvfat.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
block/vvfat.c +4 -5
@@ -1408,7 +1408,7 @@ read_cluster_directory:assert (s->current_fd); - offset=s->cluster_size*(cluster_num-s->current_mapping->begin)+s->current_mapping->info.file.offset; + offset=s->cluster_size*((cluster_num - s->current_mapping->begin) + s->current_mapping->info.file.offset);if (lseek (s->current_fd, offset,SEEK_SET )!=offset)return -3 ; s->cluster=s->cluster_buffer;
@@ -1929,8 +1929,8 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const ch (mapping->mode & MODE_DIRECTORY) == 0) {/* was modified in qcow */ -if (offset != mapping->info.file.offset + s->cluster_size - * (cluster_num - mapping->begin)) { +if (offset != s->cluster_size + * ((cluster_num - mapping->begin) + mapping->info.file.offset)) {/* offset of this cluster in file chain has changed */ abort (); copy_it =1 ;
@@ -1944,7 +1944,6 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const chif (mapping->first_mapping_index != first_mapping_index && mapping->info.file.offset >0 ) { -abort (); copy_it =1 ; }
@@ -2404,7 +2403,7 @@ static int commit_mappings(BDRVVVFATState* s, (mapping->end - mapping->begin); }else next_mapping->info.file.offset = mapping->info.file.offset + - mapping->end - mapping->begin; + (mapping->end - mapping->begin); mapping = next_mapping; }
Signature · 1 line
2.45.0
PATCH v3 3/6 vvfat: Fix reading files with non-continuous clusters
In reply to Amjad AlsharafiWhen reading with read_cluster we get the mapping with find_mapping_for_cluster and then we call open_file for this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters 500 -> 503, this will give us 2 mappings one has the range 500..501 and another 503..504, both point to the same file, but different offsets.
When we don't open the file since the path is the same, we won't assign s->current_mapping and thus accessing way out of bound of the file.
From our example above, after open_file (that didn't open anything) we will get the offset into the file with s->cluster_size*(cluster_num-s->current_mapping->begin), which will give us 0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.
--- block/vvfat.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)
block/vvfat.c +14 -7
@@ -1360,15 +1360,22 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping) {if (!mapping)return -1 ; +int new_path =1 ;if (!s->current_mapping || -strcmp (s->current_mapping->path,mapping->path)) { -/* open file */ -int fd =qemu_open_old (mapping->path, + s->current_mapping->first_mapping_index!=mapping->first_mapping_index || + (new_path =strcmp (s->current_mapping->path,mapping->path))) { + +if (new_path) { +/* open file */ +int fd =qemu_open_old (mapping->path,O_RDONLY |O_BINARY |O_LARGEFILE ); -if (fd<0 ) -return -1 ; -vvfat_close_current_file (s); - s->current_fd = fd; +if (fd<0 ) +return -1 ; +vvfat_close_current_file (s); + + s->current_fd = fd; + } +assert (s->current_fd); s->current_mapping = mapping; }return 0 ;
Signature · 1 line
2.45.0
PATCH v3 4/6 iotests: Add `vvfat` tests
In reply to Amjad AlsharafiAdded several tests to verify the implementation of the vvfat driver.
We needed a way to interact with it, so created a basic fat16.py driver that handled writing correct sectors for us.
--- tests/qemu-iotests/check | 2 +- tests/qemu-iotests/fat16.py | 507 +++++++++++++++++++++++++++++ tests/qemu-iotests/tests/vvfat | 400 +++++++++++++++++++++++ tests/qemu-iotests/tests/vvfat.out | 5 + 4 files changed, 913 insertions(+), 1 deletion(-) create mode 100644 tests/qemu-iotests/fat16.py create mode 100755 tests/qemu-iotests/tests/vvfat create mode 100755 tests/qemu-iotests/tests/vvfat.out
tests/qemu-iotests/check +1 -1
@@ -84,7 +84,7 @@ def make_argparser() -> argparse.ArgumentParser: p.set_defaults(imgfmt='raw', imgproto='file') format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2', - 'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg'] + 'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg', 'vvfat'] g_fmt = p.add_argument_group( ' image format options', 'The following options set the IMGFMT environment variable. '
tests/qemu-iotests/fat16.py new file +507 -0
507 lines not on this page · raw message
tests/qemu-iotests/tests/vvfat new file +400 -0
400 lines not on this page · raw message
tests/qemu-iotests/tests/vvfat.out new file +5 -0
@@ -0,0 +1,5 @@ +............. +---------------------------------------------------------------------- +Ran 13 tests + +OK
Signature · 1 line
2.45.0
PATCH v3 5/6 iotests: Filter out `vvfat` fmt from failing tests
In reply to Amjad Alsharafivvfat is a special format and not all tests (even generic) can run without crashing. So, added unsupported_fmt: vvfat to all failling tests.
Also added vvfat format into meson.build, vvfaat tests can be run on the block-thorough suite.
--- .gitlab-ci.d/buildtest.yml | 1 + tests/qemu-iotests/001 | 1 + tests/qemu-iotests/002 | 1 + tests/qemu-iotests/003 | 1 + tests/qemu-iotests/005 | 1 + tests/qemu-iotests/008 | 1 + tests/qemu-iotests/009 | 1 + tests/qemu-iotests/010 | 1 + tests/qemu-iotests/011 | 1 + tests/qemu-iotests/012 | 1 + tests/qemu-iotests/021 | 1 + tests/qemu-iotests/032 | 1 + tests/qemu-iotests/033 | 1 + tests/qemu-iotests/052 | 1 + tests/qemu-iotests/094 | 1 + tests/qemu-iotests/120 | 2 +- tests/qemu-iotests/140 | 1 + tests/qemu-iotests/145 | 1 + tests/qemu-iotests/157 | 1 + tests/qemu-iotests/159 | 2 +- tests/qemu-iotests/170 | 2 +- tests/qemu-iotests/192 | 1 + tests/qemu-iotests/197 | 2 +- tests/qemu-iotests/208 | 2 +- tests/qemu-iotests/215 | 2 +- tests/qemu-iotests/236 | 2 +- tests/qemu-iotests/251 | 1 + tests/qemu-iotests/307 | 2 +- tests/qemu-iotests/308 | 2 +- tests/qemu-iotests/meson.build | 3 ++- tests/qemu-iotests/tests/export-incoming-iothread | 2 +- tests/qemu-iotests/tests/fuse-allow-other | 1 + tests/qemu-iotests/tests/mirror-ready-cancel-error | 2 +- tests/qemu-iotests/tests/regression-vhdx-log | 1 + 34 files changed, 35 insertions(+), 12 deletions(-)
.gitlab-ci.d/buildtest.yml +1 -0
@@ -347,6 +347,7 @@ build-tcg-disabled: 124 132 139 142 144 145 151 152 155 157 165 194 196 200 202 208 209 216 218 227 234 246 247 248 250 254 255 257 258 260 261 262 263 264 270 272 273 277 279 image-fleecing + - ./check -vvfat vvfat build-user: extends: .native_build_job_template
tests/qemu-iotests/001 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic
tests/qemu-iotests/002 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/003 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/005 +1 -0
@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _supported_os Linux _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
tests/qemu-iotests/008 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic
tests/qemu-iotests/009 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/010 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/011 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/012 +1 -0
@@ -40,6 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto file # Remove once all tests are fixed to use TEST_IMG_FILE
tests/qemu-iotests/021 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic
tests/qemu-iotests/032 +1 -0
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This works for any image format (though unlikely to segfault for raw) _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/033 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/052 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto file # Don't do O_DIRECT on tmpfs
tests/qemu-iotests/094 +1 -0
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.qemu _supported_fmt generic +_unsupported_fmt vvfat _supported_proto nbd _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
tests/qemu-iotests/120 +1 -1
@@ -40,7 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file fuse -_unsupported_fmt luks +_unsupported_fmt luks vvfat _require_drivers raw _make_test_img 64M
tests/qemu-iotests/140 +1 -0
@@ -45,6 +45,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.qemu _supported_fmt generic +_unsupported_fmt vvfat _supported_proto file fuse _supported_os Linux
tests/qemu-iotests/145 +1 -0
@@ -39,6 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _make_test_img 1M
tests/qemu-iotests/157 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto file _require_devices virtio-blk
tests/qemu-iotests/159 +1 -1
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_unsupported_fmt luks +_unsupported_fmt luks vvfat TEST_SIZES="5 512 1024 1999 1K 64K 1M"
tests/qemu-iotests/170 +1 -1
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_unsupported_fmt luks +_unsupported_fmt luks vvfat echo echo "== Creating image =="
tests/qemu-iotests/192 +1 -0
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.qemu _supported_fmt generic +_unsupported_fmt vvfat _supported_proto file if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
tests/qemu-iotests/197 +1 -1
@@ -53,7 +53,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic # LUKS support may be possible, but it complicates things. -_unsupported_fmt luks +_unsupported_fmt luks vvfat _unsupported_imgopts "subformat=streamOptimized" echo
tests/qemu-iotests/208 +1 -1
@@ -23,7 +23,7 @@ import iotests -iotests.script_initialize(supported_fmts=['generic']) +iotests.script_initialize(supported_fmts=['generic'], unsupported_fmts=['vvfat']) with iotests.FilePath('disk.img') as disk_img_path, \ iotests.FilePath('disk-snapshot.img') as disk_snapshot_img_path, \
tests/qemu-iotests/215 +1 -1
@@ -50,7 +50,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic # LUKS support may be possible, but it complicates things. -_unsupported_fmt luks +_unsupported_fmt luks vvfat _unsupported_imgopts "subformat=streamOptimized" echo
tests/qemu-iotests/236 +1 -1
@@ -23,7 +23,7 @@ import iotests from iotests import log -iotests.script_initialize(supported_fmts=['generic']) +iotests.script_initialize(supported_fmts=['generic'], unsupported_fmts=['vvfat']) size = 64 * 1024 * 1024 granularity = 64 * 1024
tests/qemu-iotests/251 +1 -0
@@ -39,6 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.qemu _supported_fmt generic +_unsupported_fmt vvfat _supported_proto file _supported_os Linux _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/307 +1 -1
@@ -27,7 +27,7 @@ import os # luks which requires special command lines) iotests.script_initialize( supported_fmts=['generic'], - unsupported_fmts=['luks', 'vpc'], + unsupported_fmts=['luks', 'vpc', 'vvfat'], supported_platforms=['linux'], )
tests/qemu-iotests/308 +1 -1
@@ -47,7 +47,7 @@ if [ "$IMGOPTSSYNTAX" = "true" ]; then fi # We need the image to have exactly the specified size, and VPC does # not allow that by default -_unsupported_fmt vpc +_unsupported_fmt vpc vvfat _supported_proto file # We create the FUSE export manually _supported_os Linux # We need /dev/urandom
tests/qemu-iotests/meson.build +2 -1
@@ -23,7 +23,8 @@ qemu_iotests_formats = { 'raw': 'slow', 'qed': 'thorough', 'vmdk': 'thorough', - 'vpc': 'thorough' + 'vpc': 'thorough', + 'vvfat': 'thorough', } foreach k, v : emulators
tests/qemu-iotests/tests/export-incoming-iothread +1 -1
@@ -75,5 +75,5 @@ class TestExportIncomingIothread(iotests.QMPTestCase): if __name__ == '__main__': iotests.main(supported_fmts=['generic'], - unsupported_fmts=['luks'], # Would need a secret + unsupported_fmts=['luks', 'vvfat'], # Would need a secret supported_protocols=['file'])
tests/qemu-iotests/tests/fuse-allow-other +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ../common.qemu _supported_fmt generic +_unsupported_fmt vvfat _supported_proto file # We create the FUSE export manually
tests/qemu-iotests/tests/mirror-ready-cancel-error +1 -1
@@ -138,5 +138,5 @@ class TestMirrorReadyCancelError(iotests.QMPTestCase): if __name__ == '__main__': # LUKS would require special key-secret handling in add_blockdevs() iotests.main(supported_fmts=['generic'], - unsupported_fmts=['luks'], + unsupported_fmts=['luks', 'vvfat'], supported_protocols=['file'])
tests/qemu-iotests/tests/regression-vhdx-log +1 -0
@@ -40,6 +40,7 @@ cd .. . ./common.filter _supported_fmt generic +_unsupported_fmt vvfat _supported_proto generic _unsupported_imgopts "subformat=streamOptimized"
Signature · 1 line
2.45.0
Re: [PATCH v3 5/6] iotests: Filter out `vvfat` fmt from failing tests
Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:
Quoted 8 lines
vvfatis a special format and not all tests (even generic) can run without crashing. So, addedunsupported_fmt: vvfatto all failling tests.Also added
vvfatformat intomeson.build, vvfaat tests can be run on theblock-thoroughsuite.Signed-off-by Amjad Alsharafi <[email protected]>
I think the better approach is just not counting vvfat as generic. It's technically not even a format, but a protocol, though I think I agree with adding it as a format anyway because you can't store a normal image inside of it.
This should do the trick and avoid most of the changes in this patch:
tests/qemu-iotests/testenv.py +1 -1
@@ -250,7 +250,7 @@ def __init__(self, source_dir: str, build_dir: str, self.qemu_img_options = os.getenv('QEMU_IMG_OPTIONS') self.qemu_nbd_options = os.getenv('QEMU_NBD_OPTIONS') - is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg'] + is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg', 'vvfat'] self.imgfmt_generic = 'true' if is_generic else 'false' self.qemu_io_options = f'--cache {self.cachemode} --aio {self.aiomode}'
Kevin
PATCH v3 6/6 iotests: Add `create_file` test for `vvfat` driver
In reply to Amjad AlsharafiWe test the ability to create new files in the filesystem, this is done by adding an entry in the desired directory list.
The file will also be created in the host filesystem with matching filename.
--- tests/qemu-iotests/fat16.py | 124 +++++++++++++++++++++++++++-- tests/qemu-iotests/tests/vvfat | 29 +++++-- tests/qemu-iotests/tests/vvfat.out | 4 +- 3 files changed, 144 insertions(+), 13 deletions(-)
tests/qemu-iotests/fat16.py +118 -6
@@ -16,9 +16,11 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. from typing import List +import string SECTOR_SIZE = 512 DIRENTRY_SIZE = 32 +ALLOWED_FILE_CHARS = set("!#$%&'()-@^_`{}~" + string.digits + string.ascii_uppercase) class MBR:
@@ -265,7 +267,7 @@ def write_fat_entry(self, cluster: int, value: int): + self.fats[fat_offset + 2 :] ) self.fats_dirty_sectors.add(fat_offset // SECTOR_SIZE) - + def flush_fats(self): """ Write the FATs back to the disk.
@@ -293,7 +295,7 @@ def next_cluster(self, cluster: int) -> int | None: raise Exception("Invalid FAT entry") else: return fat_entry - + def next_free_cluster(self) -> int: """ Find the next free cluster.
@@ -338,6 +340,67 @@ def read_directory(self, cluster: int) -> List[FatDirectoryEntry]: cluster = self.next_cluster(cluster) return entries + def add_direntry(self, cluster: int | None, name: str, ext: str, attributes: int): + """ + Add a new directory entry to the given cluster. + If the cluster is `None`, then it will be added to the root directory. + """ + + def find_free_entry(data: bytes): + for i in range(0, len(data), DIRENTRY_SIZE): + entry = data[i : i + DIRENTRY_SIZE] + if entry[0] == 0 or entry[0] == 0xE5: + return i + return None + + assert len(name) <= 8, "Name must be 8 characters or less" + assert len(ext) <= 3, "Ext must be 3 characters or less" + assert attributes % 0x15 != 0x15, "Invalid attributes" + + # initial dummy data + new_entry = FatDirectoryEntry(b"\0" * 32, 0, 0) + new_entry.name = name.ljust(8, " ") + new_entry.ext = ext.ljust(3, " ") + new_entry.attributes = attributes + new_entry.reserved = 0 + new_entry.create_time_tenth = 0 + new_entry.create_time = 0 + new_entry.create_date = 0 + new_entry.last_access_date = 0 + new_entry.last_mod_time = 0 + new_entry.last_mod_date = 0 + new_entry.cluster = self.next_free_cluster() + new_entry.size_bytes = 0 + + # mark as EOF + self.write_fat_entry(new_entry.cluster, 0xFFFF) + + if cluster is None: + for i in range(self.boot_sector.root_dir_size()): + sector_data = self.read_sectors( + self.boot_sector.root_dir_start() + i, 1 + ) + offset = find_free_entry(sector_data) + if offset is not None: + new_entry.sector = self.boot_sector.root_dir_start() + i + new_entry.offset = offset + self.update_direntry(new_entry) + return new_entry + else: + while cluster is not None: + data = self.read_cluster(cluster) + offset = find_free_entry(data) + if offset is not None: + new_entry.sector = self.boot_sector.first_sector_of_cluster( + cluster + ) + (offset // SECTOR_SIZE) + new_entry.offset = offset % SECTOR_SIZE + self.update_direntry(new_entry) + return new_entry + cluster = self.next_cluster(cluster) + + raise Exception("No free directory entries") + def update_direntry(self, entry: FatDirectoryEntry): """ Write the directory entry back to the disk.
@@ -406,9 +469,10 @@ def truncate_file(self, entry: FatDirectoryEntry, new_size: int): raise Exception(f"{entry.whole_name()} is a directory") def clusters_from_size(size: int): - return (size + self.boot_sector.cluster_bytes() - 1) // self.boot_sector.cluster_bytes() + return ( + size + self.boot_sector.cluster_bytes() - 1 + ) // self.boot_sector.cluster_bytes() - # First, allocate new FATs if we need to required_clusters = clusters_from_size(new_size) current_clusters = clusters_from_size(entry.size_bytes)
@@ -438,7 +502,7 @@ def clusters_from_size(size: int): self.write_fat_entry(cluster, new_cluster) self.write_fat_entry(new_cluster, 0xFFFF) cluster = new_cluster - + elif required_clusters < current_clusters: # Truncate the file cluster = entry.cluster
@@ -464,7 +528,9 @@ def clusters_from_size(size: int): count += 1 affected_clusters.add(cluster) cluster = self.next_cluster(cluster) - assert count == required_clusters, f"Expected {required_clusters} clusters, got {count}" + assert ( + count == required_clusters + ), f"Expected {required_clusters} clusters, got {count}" # update the size entry.size_bytes = new_size
@@ -505,3 +571,49 @@ def write_file(self, entry: FatDirectoryEntry, data: bytes): cluster = self.next_cluster(cluster) assert len(data) == 0, "Data was not written completely, clusters missing" + + def create_file(self, path: str): + """ + Create a new file at the given path. + """ + assert path[0] == "/", "Path must start with /" + + path = path[1:] # remove the leading / + + parts = path.split("/") + + directory_cluster = None + directory = self.read_root_directory() + + parts, filename = parts[:-1], parts[-1] + + for i, part in enumerate(parts): + current_entry = None + for entry in directory: + if entry.whole_name() == part: + current_entry = entry + break + if current_entry is None: + return None + + if current_entry.attributes & 0x10 == 0: + raise Exception(f"{current_entry.whole_name()} is not a directory") + else: + directory = self.read_directory(current_entry.cluster) + directory_cluster = current_entry.cluster + + # add new entry to the directory + + filename, ext = filename.split(".") + + if len(ext) > 3: + raise Exception("Ext must be 3 characters or less") + if len(filename) > 8: + raise Exception("Name must be 8 characters or less") + + for c in filename + ext: + + if c not in ALLOWED_FILE_CHARS: + raise Exception("Invalid character in filename") + + return self.add_direntry(directory_cluster, filename, ext, 0)
tests/qemu-iotests/tests/vvfat +24 -5
@@ -323,7 +323,7 @@ class TestVVFatDriver(QMPTestCase): with open(os.path.join(filesystem, "file0.txt"), "rb") as f: self.assertEqual(f.read(), new_content) - + def test_write_large_file(self): """ Test writing a large file
@@ -342,7 +342,7 @@ class TestVVFatDriver(QMPTestCase): with open(os.path.join(filesystem, "large1.txt"), "rb") as f: self.assertEqual(f.read(), new_content) - + def test_truncate_file_change_clusters_less(self): """ Test truncating a file by reducing the number of clusters
@@ -359,7 +359,6 @@ class TestVVFatDriver(QMPTestCase): with open(os.path.join(filesystem, "large1.txt"), "rb") as f: self.assertEqual(f.read(), b"A") - def test_write_file_change_clusters_less(self): """ Test truncating a file by reducing the number of clusters
@@ -376,7 +375,7 @@ class TestVVFatDriver(QMPTestCase): with open(os.path.join(filesystem, "large2.txt"), "rb") as f: self.assertEqual(f.read(), new_content) - + def test_write_file_change_clusters_more(self): """ Test truncating a file by increasing the number of clusters
@@ -392,7 +391,27 @@ class TestVVFatDriver(QMPTestCase): with open(os.path.join(filesystem, "large2.txt"), "rb") as f: self.assertEqual(f.read(), new_content) - + + def test_create_file(self): + """ + Test creating a new file + """ + fat16 = self.init_fat16() + + new_file = fat16.create_file("/NEWFILE.TXT") + + self.assertIsNotNone(new_file) + self.assertEqual(new_file.size_bytes, 0) + + new_content = b"Hello, world! New file\n" + fat16.write_file(new_file, new_content) + + self.assertEqual(fat16.read_file(new_file), new_content) + + with open(os.path.join(filesystem, "newfile.txt"), "rb") as f: + self.assertEqual(f.read(), new_content) + + # TODO: support deleting files if __name__ == "__main__":
tests/qemu-iotests/tests/vvfat.out +2 -2
@@ -1,5 +1,5 @@ -............. +.............. ---------------------------------------------------------------------- -Ran 13 tests +Ran 14 tests OK
Signature · 1 line
2.45.0
Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:
Quoted 13 lines
These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.
Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash onabort()when the file is modified and a new cluster was added.
Quoted 4 lines
Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.
I was wondering how to reproduce the bugs that patches 2 and 3 fix. So I tried to run your iotests case, and while it does catch the bug that patch 1 fixes, it passes even without the other two fixes.
Is this expected? If so, can we add more tests that trigger the problems the other two patches address?
Kevin
Quoted 19 lines · Kevin Wolf
Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:
Quoted 13 lines
These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.
Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash onabort()when the file is modified and a new cluster was added.Quoted 4 lines
Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.
Quoted 9 lines
I was wondering how to reproduce the bugs that patches 2 and 3 fix. So I tried to run your iotests case, and while it does catch the bug that patch 1 fixes, it passes even without the other two fixes.
Is this expected? If so, can we add more tests that trigger the problems the other two patches address?
Kevin
Thanks for checking, so this bug happens when you have mapping for file, and the clusters are not contiguous.
For example, a file with clusters 12, 13, 15, here when trying to read from cluster 15, it will get the offset in the file by using the formula cluster_size * (15-12) (12 is the first cluster).
This is of course is not correct, and will result in error reading the file from outside the range.
The reason it wasn't clear when you tested it, is that since I'm modifying large2.txt, and its the last file in the disk, when trying to allocate new clusters, coincidentally, the new clusters are allocated after the last cluster of that same file, so the issue wasn't triggered.
I'll modify the test to use the other file, so that we can trigger the issue.
I'll also modify the other suggestions you had in the other patches and submit a new version.
Amjad