Skip to contentSkip to conversation

vvfat: Fix write bugs for large files and add iotests

PATCH Superseded Amjad Alsharafi 2 participants 11 messages May 26, 2024 – Jun 5, 2024

Subsystems (2)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover vvfat: Fix write bugs for large files and add iotests

These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.

Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash on abort() when the file is modified and a new cluster was added.

Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.

v3:
Added test for creating new files in vvfat.

v2:
Added iotests for vvfat driver along with a simple fat16 module to run the tests.

v1:
https://patchew.org/QEMU/[email protected]/
Fix the issue of writing to the middle of the file in vvfat

Amjad Alsharafi (6):
vvfat: Fix bug in writing to middle of file
vvfat: Fix usage of info.file.offset
vvfat: Fix reading files with non-continuous clusters
iotests: Add vvfat tests
iotests: Filter out vvfat fmt from failing tests
iotests: Add create_file test for vvfat driver

 .gitlab-ci.d/buildtest.yml                    |   1 +
 block/vvfat.c                                 |  32 +-
 tests/qemu-iotests/001                        |   1 +
 tests/qemu-iotests/002                        |   1 +
 tests/qemu-iotests/003                        |   1 +
 tests/qemu-iotests/005                        |   1 +
 tests/qemu-iotests/008                        |   1 +
 tests/qemu-iotests/009                        |   1 +
 tests/qemu-iotests/010                        |   1 +
 tests/qemu-iotests/011                        |   1 +
 tests/qemu-iotests/012                        |   1 +
 tests/qemu-iotests/021                        |   1 +
 tests/qemu-iotests/032                        |   1 +
 tests/qemu-iotests/033                        |   1 +
 tests/qemu-iotests/052                        |   1 +
 tests/qemu-iotests/094                        |   1 +
 tests/qemu-iotests/120                        |   2 +-
 tests/qemu-iotests/140                        |   1 +
 tests/qemu-iotests/145                        |   1 +
 tests/qemu-iotests/157                        |   1 +
 tests/qemu-iotests/159                        |   2 +-
 tests/qemu-iotests/170                        |   2 +-
 tests/qemu-iotests/192                        |   1 +
 tests/qemu-iotests/197                        |   2 +-
 tests/qemu-iotests/208                        |   2 +-
 tests/qemu-iotests/215                        |   2 +-
 tests/qemu-iotests/236                        |   2 +-
 tests/qemu-iotests/251                        |   1 +
 tests/qemu-iotests/307                        |   2 +-
 tests/qemu-iotests/308                        |   2 +-
 tests/qemu-iotests/check                      |   2 +-
 tests/qemu-iotests/fat16.py                   | 619 ++++++++++++++++++
 tests/qemu-iotests/meson.build                |   3 +-
 .../tests/export-incoming-iothread            |   2 +-
 tests/qemu-iotests/tests/fuse-allow-other     |   1 +
 .../tests/mirror-ready-cancel-error           |   2 +-
 tests/qemu-iotests/tests/regression-vhdx-log  |   1 +
 tests/qemu-iotests/tests/vvfat                | 419 ++++++++++++
 tests/qemu-iotests/tests/vvfat.out            |   5 +
 39 files changed, 1098 insertions(+), 26 deletions(-)
 create mode 100644 tests/qemu-iotests/fat16.py
 create mode 100755 tests/qemu-iotests/tests/vvfat
 create mode 100755 tests/qemu-iotests/tests/vvfat.out
Signature · 1 line
2.45.0
#1 lore raw

PATCH v3 1/6 vvfat: Fix bug in writing to middle of file

Before this commit, the behavior when calling commit_one_file for example with offset=0x2000 (second cluster), what will happen is that we won't fetch the next cluster from the fat, and instead use the first cluster for the read operation.

This is due to off-by-one error here, where i=0x2000 !< offset=0x2000, thus not fetching the next cluster.

---
 block/vvfat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/vvfat.c +1 -1
index 9d050ba3ae..ab342f0743 100644
@@ -2525,7 +2525,7 @@ commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset)
         return -1;
     }
 
-    for (i = s->cluster_size; i < offset; i += s->cluster_size)
+    for (i = s->cluster_size; i <= offset; i += s->cluster_size)
         c = modified_fat_get(s, c);
 
     fd = qemu_open_old(mapping->path, O_RDWR | O_CREAT | O_BINARY, 0666);
Signature · 1 line
2.45.0
Reviewed-by Tested-by on v3 1/6
#2 lore raw

Re: [PATCH v3 1/6] vvfat: Fix bug in writing to middle of file

Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:

Quoted 9 lines

Before this commit, the behavior when calling commit_one_file for example with offset=0x2000 (second cluster), what will happen is that we won't fetch the next cluster from the fat, and instead use the first cluster for the read operation.

This is due to off-by-one error here, where i=0x2000 !< offset=0x2000, thus not fetching the next cluster.

block/vvfat.c commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset) · line 2529

Quoted 11 lines
block/vvfat.c +1 -1
index 9d050ba3ae..ab342f0743 100644
@@ -2525,7 +2525,7 @@ commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset)
         return -1;
     }
 
-    for (i = s->cluster_size; i < offset; i += s->cluster_size)
+    for (i = s->cluster_size; i <= offset; i += s->cluster_size)
         c = modified_fat_get(s, c);

While your change results in the correct behaviour, I think I would prefer the code to be changed like this so that at the start of each loop iteration, 'i' always refers to the offset that matches 'c':

    for (i = 0; i < offset; i += s->cluster_size) {
        c = modified_fat_get(s, c);
    }

I'm also adding braces here to make the code conform with the QEMU coding style. You can use scripts/checkpatch.pl to make sure that all code you add has the correct style. Much of the vvfat code predates the coding style, so you'll often have to change style when you touch a line. (Which is good, because it slowly fixes the existing mess.)

You can keep my Reviewed/Tested-by if you change this.

Kevin

#3 lore raw

PATCH v3 2/6 vvfat: Fix usage of `info.file.offset`

In reply to Amjad Alsharafi

The field is marked as "the offset in the file (in clusters)", but it was being used like this cluster_size*(nums)+mapping->info.file.offset, which is incorrect.

Additionally, removed the abort when first_mapping_index does not match, as this matches the case when adding new clusters for files, and its inevitable that we reach this condition when doing that if the clusters are not after one another, so there is no reason to abort here, execution continues and the new clusters are written to disk correctly.

---
 block/vvfat.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
block/vvfat.c +4 -5
index ab342f0743..cb3ab81e29 100644
@@ -1408,7 +1408,7 @@ read_cluster_directory:
 
         assert(s->current_fd);
 
-        offset=s->cluster_size*(cluster_num-s->current_mapping->begin)+s->current_mapping->info.file.offset;
+        offset=s->cluster_size*((cluster_num - s->current_mapping->begin) + s->current_mapping->info.file.offset);
         if(lseek(s->current_fd, offset, SEEK_SET)!=offset)
             return -3;
         s->cluster=s->cluster_buffer;
@@ -1929,8 +1929,8 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const ch
                         (mapping->mode & MODE_DIRECTORY) == 0) {
 
                     /* was modified in qcow */
-                    if (offset != mapping->info.file.offset + s->cluster_size
-                            * (cluster_num - mapping->begin)) {
+                    if (offset != s->cluster_size
+                            * ((cluster_num - mapping->begin) + mapping->info.file.offset)) {
                         /* offset of this cluster in file chain has changed */
                         abort();
                         copy_it = 1;
@@ -1944,7 +1944,6 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const ch
 
                     if (mapping->first_mapping_index != first_mapping_index
                             && mapping->info.file.offset > 0) {
-                        abort();
                         copy_it = 1;
                     }
 
@@ -2404,7 +2403,7 @@ static int commit_mappings(BDRVVVFATState* s,
                         (mapping->end - mapping->begin);
             } else
                 next_mapping->info.file.offset = mapping->info.file.offset +
-                        mapping->end - mapping->begin;
+                        (mapping->end - mapping->begin);
 
             mapping = next_mapping;
         }
Signature · 1 line
2.45.0
#4 lore raw

PATCH v3 3/6 vvfat: Fix reading files with non-continuous clusters

In reply to Amjad Alsharafi

When reading with read_cluster we get the mapping with find_mapping_for_cluster and then we call open_file for this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters 500 -> 503, this will give us 2 mappings one has the range 500..501 and another 503..504, both point to the same file, but different offsets.

When we don't open the file since the path is the same, we won't assign s->current_mapping and thus accessing way out of bound of the file.

From our example above, after open_file (that didn't open anything) we will get the offset into the file with s->cluster_size*(cluster_num-s->current_mapping->begin), which will give us 0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.

---
 block/vvfat.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
block/vvfat.c +14 -7
index cb3ab81e29..87165abc26 100644
@@ -1360,15 +1360,22 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping)
 {
     if(!mapping)
         return -1;
+    int new_path = 1;
     if(!s->current_mapping ||
-            strcmp(s->current_mapping->path,mapping->path)) {
-        /* open file */
-        int fd = qemu_open_old(mapping->path,
+            s->current_mapping->first_mapping_index!=mapping->first_mapping_index ||
+            (new_path = strcmp(s->current_mapping->path,mapping->path))) {
+
+        if (new_path) {
+            /* open file */
+            int fd = qemu_open_old(mapping->path,
                                O_RDONLY | O_BINARY | O_LARGEFILE);
-        if(fd<0)
-            return -1;
-        vvfat_close_current_file(s);
-        s->current_fd = fd;
+            if(fd<0)
+                return -1;
+            vvfat_close_current_file(s);
+
+            s->current_fd = fd;
+        }
+        assert(s->current_fd);
         s->current_mapping = mapping;
     }
     return 0;
Signature · 1 line
2.45.0
#5 lore raw

PATCH v3 4/6 iotests: Add `vvfat` tests

In reply to Amjad Alsharafi

Added several tests to verify the implementation of the vvfat driver.

We needed a way to interact with it, so created a basic fat16.py driver that handled writing correct sectors for us.

---
 tests/qemu-iotests/check           |   2 +-
 tests/qemu-iotests/fat16.py        | 507 +++++++++++++++++++++++++++++
 tests/qemu-iotests/tests/vvfat     | 400 +++++++++++++++++++++++
 tests/qemu-iotests/tests/vvfat.out |   5 +
 4 files changed, 913 insertions(+), 1 deletion(-)
 create mode 100644 tests/qemu-iotests/fat16.py
 create mode 100755 tests/qemu-iotests/tests/vvfat
 create mode 100755 tests/qemu-iotests/tests/vvfat.out
tests/qemu-iotests/check +1 -1
index 56d88ca423..545f9ec7bd 100755
@@ -84,7 +84,7 @@ def make_argparser() -> argparse.ArgumentParser:
     p.set_defaults(imgfmt='raw', imgproto='file')
 
     format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2',
-                   'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg']
+                   'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg', 'vvfat']
     g_fmt = p.add_argument_group(
         '  image format options',
         'The following options set the IMGFMT environment variable. '
tests/qemu-iotests/fat16.py new file +507 -0

507 lines not on this page · raw message

tests/qemu-iotests/tests/vvfat new file +400 -0

400 lines not on this page · raw message

tests/qemu-iotests/tests/vvfat.out new file +5 -0
new file mode 100755
index 0000000000..fa16b5ccef
@@ -0,0 +1,5 @@
+.............
+----------------------------------------------------------------------
+Ran 13 tests
+
+OK
Signature · 1 line
2.45.0
#6 lore raw

PATCH v3 5/6 iotests: Filter out `vvfat` fmt from failing tests

In reply to Amjad Alsharafi

vvfat is a special format and not all tests (even generic) can run without crashing. So, added unsupported_fmt: vvfat to all failling tests.

Also added vvfat format into meson.build, vvfaat tests can be run on the block-thorough suite.

---
 .gitlab-ci.d/buildtest.yml                         | 1 +
 tests/qemu-iotests/001                             | 1 +
 tests/qemu-iotests/002                             | 1 +
 tests/qemu-iotests/003                             | 1 +
 tests/qemu-iotests/005                             | 1 +
 tests/qemu-iotests/008                             | 1 +
 tests/qemu-iotests/009                             | 1 +
 tests/qemu-iotests/010                             | 1 +
 tests/qemu-iotests/011                             | 1 +
 tests/qemu-iotests/012                             | 1 +
 tests/qemu-iotests/021                             | 1 +
 tests/qemu-iotests/032                             | 1 +
 tests/qemu-iotests/033                             | 1 +
 tests/qemu-iotests/052                             | 1 +
 tests/qemu-iotests/094                             | 1 +
 tests/qemu-iotests/120                             | 2 +-
 tests/qemu-iotests/140                             | 1 +
 tests/qemu-iotests/145                             | 1 +
 tests/qemu-iotests/157                             | 1 +
 tests/qemu-iotests/159                             | 2 +-
 tests/qemu-iotests/170                             | 2 +-
 tests/qemu-iotests/192                             | 1 +
 tests/qemu-iotests/197                             | 2 +-
 tests/qemu-iotests/208                             | 2 +-
 tests/qemu-iotests/215                             | 2 +-
 tests/qemu-iotests/236                             | 2 +-
 tests/qemu-iotests/251                             | 1 +
 tests/qemu-iotests/307                             | 2 +-
 tests/qemu-iotests/308                             | 2 +-
 tests/qemu-iotests/meson.build                     | 3 ++-
 tests/qemu-iotests/tests/export-incoming-iothread  | 2 +-
 tests/qemu-iotests/tests/fuse-allow-other          | 1 +
 tests/qemu-iotests/tests/mirror-ready-cancel-error | 2 +-
 tests/qemu-iotests/tests/regression-vhdx-log       | 1 +
 34 files changed, 35 insertions(+), 12 deletions(-)
.gitlab-ci.d/buildtest.yml +1 -0
index cfdff175c3..a46c179a6b 100644
@@ -347,6 +347,7 @@ build-tcg-disabled:
             124 132 139 142 144 145 151 152 155 157 165 194 196 200 202
             208 209 216 218 227 234 246 247 248 250 254 255 257 258
             260 261 262 263 264 270 272 273 277 279 image-fleecing
+    - ./check -vvfat vvfat
 
 build-user:
   extends: .native_build_job_template
tests/qemu-iotests/001 +1 -0
index 6f980fd34d..cf905b5d00 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 
 
tests/qemu-iotests/002 +1 -0
index 5ce1647531..1e557fad8c 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
tests/qemu-iotests/003 +1 -0
index 03f902a83c..6e74f1faeb 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
tests/qemu-iotests/005 +1 -0
index ba377543b0..28ae66bfcd 100755
@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _supported_os Linux
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
tests/qemu-iotests/008 +1 -0
index fa4990b513..80850ecf12 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 
 
tests/qemu-iotests/009 +1 -0
index efa852bad3..408617b0bc 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
tests/qemu-iotests/010 +1 -0
index 4ae9027b47..c9f6279255 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
tests/qemu-iotests/011 +1 -0
index 5c99ac987f..92039fa949 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
tests/qemu-iotests/012 +1 -0
index 3a24d2ca8d..5b0f1338e6 100755
@@ -40,6 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto file
 
 # Remove once all tests are fixed to use TEST_IMG_FILE
tests/qemu-iotests/021 +1 -0
index 0fc89df2fe..475f9b2116 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 
 
tests/qemu-iotests/032 +1 -0
index ebbe7cb0ba..b58141f132 100755
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 # This works for any image format (though unlikely to segfault for raw)
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
tests/qemu-iotests/033 +1 -0
index 4bc7a071bd..6410c8717e 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
tests/qemu-iotests/052 +1 -0
index 2f23ac9b65..5b3545c8b9 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto file
 
 # Don't do O_DIRECT on tmpfs
tests/qemu-iotests/094 +1 -0
index 4766e9a458..d8da955c1b 100755
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.qemu
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto nbd
 _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
 
tests/qemu-iotests/120 +1 -1
index ac7bd8c4e3..d8e5f4241a 100755
@@ -40,7 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto file fuse
-_unsupported_fmt luks
+_unsupported_fmt luks vvfat
 _require_drivers raw
 
 _make_test_img 64M
tests/qemu-iotests/140 +1 -0
index d923b777e2..42a96d9097 100755
@@ -45,6 +45,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.qemu
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto file fuse
 _supported_os Linux
 
tests/qemu-iotests/145 +1 -0
index a2ce92516d..ff9c6ff54f 100755
@@ -39,6 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 
 _make_test_img 1M
tests/qemu-iotests/157 +1 -0
index aa2ebbfb4b..419d3b8b7a 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto file
 
 _require_devices virtio-blk
tests/qemu-iotests/159 +1 -1
index 4eb476d3a8..70a1079ae5 100755
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto file
-_unsupported_fmt luks
+_unsupported_fmt luks vvfat
 
 TEST_SIZES="5 512 1024 1999 1K 64K 1M"
 
tests/qemu-iotests/170 +1 -1
index 41387e4d66..f08fb0e8bd 100755
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto file
-_unsupported_fmt luks
+_unsupported_fmt luks vvfat
 
 echo
 echo "== Creating image =="
tests/qemu-iotests/192 +1 -0
index e66e1a4f06..ca72b0b7c8 100755
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.qemu
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto file
 
 if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
tests/qemu-iotests/197 +1 -1
index 69849c800e..76b30672d9 100755
@@ -53,7 +53,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
-_unsupported_fmt luks
+_unsupported_fmt luks vvfat
 _unsupported_imgopts "subformat=streamOptimized"
 
 echo
tests/qemu-iotests/208 +1 -1
index 6117f165fa..f08c83c0c1 100755
@@ -23,7 +23,7 @@
 
 import iotests
 
-iotests.script_initialize(supported_fmts=['generic'])
+iotests.script_initialize(supported_fmts=['generic'], unsupported_fmts=['vvfat'])
 
 with iotests.FilePath('disk.img') as disk_img_path, \
      iotests.FilePath('disk-snapshot.img') as disk_snapshot_img_path, \
tests/qemu-iotests/215 +1 -1
index 6babbcdc1f..3bd03c741e 100755
@@ -50,7 +50,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
-_unsupported_fmt luks
+_unsupported_fmt luks vvfat
 _unsupported_imgopts "subformat=streamOptimized"
 
 echo
tests/qemu-iotests/236 +1 -1
index 20419bbb9e..4bcca355ab 100755
@@ -23,7 +23,7 @@
 import iotests
 from iotests import log
 
-iotests.script_initialize(supported_fmts=['generic'])
+iotests.script_initialize(supported_fmts=['generic'], unsupported_fmts=['vvfat'])
 size = 64 * 1024 * 1024
 granularity = 64 * 1024
 
tests/qemu-iotests/251 +1 -0
index 794cad58b2..ac83f69d9a 100755
@@ -39,6 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.qemu
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=streamOptimized"
tests/qemu-iotests/307 +1 -1
index b429b5aa50..548d2f040d 100755
@@ -27,7 +27,7 @@ import os
 # luks which requires special command lines)
 iotests.script_initialize(
     supported_fmts=['generic'],
-    unsupported_fmts=['luks', 'vpc'],
+    unsupported_fmts=['luks', 'vpc', 'vvfat'],
     supported_platforms=['linux'],
 )
 
tests/qemu-iotests/308 +1 -1
index ea81dc496a..993dd2a5f9 100755
@@ -47,7 +47,7 @@ if [ "$IMGOPTSSYNTAX" = "true" ]; then
 fi
 # We need the image to have exactly the specified size, and VPC does
 # not allow that by default
-_unsupported_fmt vpc
+_unsupported_fmt vpc vvfat
 
 _supported_proto file # We create the FUSE export manually
 _supported_os Linux # We need /dev/urandom
tests/qemu-iotests/meson.build +2 -1
index fad340ad59..e87cf71fc4 100644
@@ -23,7 +23,8 @@ qemu_iotests_formats = {
   'raw': 'slow',
   'qed': 'thorough',
   'vmdk': 'thorough',
-  'vpc': 'thorough'
+  'vpc': 'thorough',
+  'vvfat': 'thorough',
 }
 
 foreach k, v : emulators
tests/qemu-iotests/tests/export-incoming-iothread +1 -1
index d36d6194e0..9535046dfd 100755
@@ -75,5 +75,5 @@ class TestExportIncomingIothread(iotests.QMPTestCase):
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['generic'],
-                 unsupported_fmts=['luks'], # Would need a secret
+                 unsupported_fmts=['luks', 'vvfat'], # Would need a secret
                  supported_protocols=['file'])
tests/qemu-iotests/tests/fuse-allow-other +1 -0
index 19f494aefb..6cfbe9ef1f 100755
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ../common.qemu
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 
 _supported_proto file # We create the FUSE export manually
 
tests/qemu-iotests/tests/mirror-ready-cancel-error +1 -1
index ed2e46447e..3b36764ecb 100755
@@ -138,5 +138,5 @@ class TestMirrorReadyCancelError(iotests.QMPTestCase):
 if __name__ == '__main__':
     # LUKS would require special key-secret handling in add_blockdevs()
     iotests.main(supported_fmts=['generic'],
-                 unsupported_fmts=['luks'],
+                 unsupported_fmts=['luks', 'vvfat'],
                  supported_protocols=['file'])
tests/qemu-iotests/tests/regression-vhdx-log +1 -0
index ca264e93d6..eb216c27dd 100755
@@ -40,6 +40,7 @@ cd ..
 . ./common.filter
 
 _supported_fmt generic
+_unsupported_fmt vvfat
 _supported_proto generic
 _unsupported_imgopts "subformat=streamOptimized"
 
Signature · 1 line
2.45.0
#7 lore raw

Re: [PATCH v3 5/6] iotests: Filter out `vvfat` fmt from failing tests

Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:

Quoted 8 lines

vvfat is a special format and not all tests (even generic) can run without crashing. So, added unsupported_fmt: vvfat to all failling tests.

Also added vvfat format into meson.build, vvfaat tests can be run on the block-thorough suite.

I think the better approach is just not counting vvfat as generic. It's technically not even a format, but a protocol, though I think I agree with adding it as a format anyway because you can't store a normal image inside of it.

This should do the trick and avoid most of the changes in this patch:

tests/qemu-iotests/testenv.py +1 -1
index 588f30a4f1..4053d29de4 100644
@@ -250,7 +250,7 @@ def __init__(self, source_dir: str, build_dir: str,
         self.qemu_img_options = os.getenv('QEMU_IMG_OPTIONS')
         self.qemu_nbd_options = os.getenv('QEMU_NBD_OPTIONS')
 
-        is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg']
+        is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg', 'vvfat']
         self.imgfmt_generic = 'true' if is_generic else 'false'
 
         self.qemu_io_options = f'--cache {self.cachemode} --aio {self.aiomode}'

Kevin

#8 lore raw

PATCH v3 6/6 iotests: Add `create_file` test for `vvfat` driver

In reply to Amjad Alsharafi

We test the ability to create new files in the filesystem, this is done by adding an entry in the desired directory list.
The file will also be created in the host filesystem with matching filename.

---
 tests/qemu-iotests/fat16.py        | 124 +++++++++++++++++++++++++++--
 tests/qemu-iotests/tests/vvfat     |  29 +++++--
 tests/qemu-iotests/tests/vvfat.out |   4 +-
 3 files changed, 144 insertions(+), 13 deletions(-)
tests/qemu-iotests/fat16.py +118 -6
index 6ac5508d8d..e86bdd0b10 100644
@@ -16,9 +16,11 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 from typing import List
+import string
 
 SECTOR_SIZE = 512
 DIRENTRY_SIZE = 32
+ALLOWED_FILE_CHARS = set("!#$%&'()-@^_`{}~" + string.digits + string.ascii_uppercase)
 
 
 class MBR:
@@ -265,7 +267,7 @@ def write_fat_entry(self, cluster: int, value: int):
             + self.fats[fat_offset + 2 :]
         )
         self.fats_dirty_sectors.add(fat_offset // SECTOR_SIZE)
-    
+
     def flush_fats(self):
         """
         Write the FATs back to the disk.
@@ -293,7 +295,7 @@ def next_cluster(self, cluster: int) -> int | None:
             raise Exception("Invalid FAT entry")
         else:
             return fat_entry
-    
+
     def next_free_cluster(self) -> int:
         """
         Find the next free cluster.
@@ -338,6 +340,67 @@ def read_directory(self, cluster: int) -> List[FatDirectoryEntry]:
             cluster = self.next_cluster(cluster)
         return entries
 
+    def add_direntry(self, cluster: int | None, name: str, ext: str, attributes: int):
+        """
+        Add a new directory entry to the given cluster.
+        If the cluster is `None`, then it will be added to the root directory.
+        """
+
+        def find_free_entry(data: bytes):
+            for i in range(0, len(data), DIRENTRY_SIZE):
+                entry = data[i : i + DIRENTRY_SIZE]
+                if entry[0] == 0 or entry[0] == 0xE5:
+                    return i
+            return None
+
+        assert len(name) <= 8, "Name must be 8 characters or less"
+        assert len(ext) <= 3, "Ext must be 3 characters or less"
+        assert attributes % 0x15 != 0x15, "Invalid attributes"
+
+        # initial dummy data
+        new_entry = FatDirectoryEntry(b"\0" * 32, 0, 0)
+        new_entry.name = name.ljust(8, " ")
+        new_entry.ext = ext.ljust(3, " ")
+        new_entry.attributes = attributes
+        new_entry.reserved = 0
+        new_entry.create_time_tenth = 0
+        new_entry.create_time = 0
+        new_entry.create_date = 0
+        new_entry.last_access_date = 0
+        new_entry.last_mod_time = 0
+        new_entry.last_mod_date = 0
+        new_entry.cluster = self.next_free_cluster()
+        new_entry.size_bytes = 0
+
+        # mark as EOF
+        self.write_fat_entry(new_entry.cluster, 0xFFFF)
+
+        if cluster is None:
+            for i in range(self.boot_sector.root_dir_size()):
+                sector_data = self.read_sectors(
+                    self.boot_sector.root_dir_start() + i, 1
+                )
+                offset = find_free_entry(sector_data)
+                if offset is not None:
+                    new_entry.sector = self.boot_sector.root_dir_start() + i
+                    new_entry.offset = offset
+                    self.update_direntry(new_entry)
+                    return new_entry
+        else:
+            while cluster is not None:
+                data = self.read_cluster(cluster)
+                offset = find_free_entry(data)
+                if offset is not None:
+                    new_entry.sector = self.boot_sector.first_sector_of_cluster(
+                        cluster
+                    ) + (offset // SECTOR_SIZE)
+                    new_entry.offset = offset % SECTOR_SIZE
+                    self.update_direntry(new_entry)
+                    return new_entry
+                cluster = self.next_cluster(cluster)
+
+        raise Exception("No free directory entries")
+
     def update_direntry(self, entry: FatDirectoryEntry):
         """
         Write the directory entry back to the disk.
@@ -406,9 +469,10 @@ def truncate_file(self, entry: FatDirectoryEntry, new_size: int):
             raise Exception(f"{entry.whole_name()} is a directory")
 
         def clusters_from_size(size: int):
-            return (size + self.boot_sector.cluster_bytes() - 1) // self.boot_sector.cluster_bytes()
+            return (
+                size + self.boot_sector.cluster_bytes() - 1
+            ) // self.boot_sector.cluster_bytes()
 
-        
         # First, allocate new FATs if we need to
         required_clusters = clusters_from_size(new_size)
         current_clusters = clusters_from_size(entry.size_bytes)
@@ -438,7 +502,7 @@ def clusters_from_size(size: int):
                 self.write_fat_entry(cluster, new_cluster)
                 self.write_fat_entry(new_cluster, 0xFFFF)
                 cluster = new_cluster
-            
+
         elif required_clusters < current_clusters:
             # Truncate the file
             cluster = entry.cluster
@@ -464,7 +528,9 @@ def clusters_from_size(size: int):
             count += 1
             affected_clusters.add(cluster)
             cluster = self.next_cluster(cluster)
-        assert count == required_clusters, f"Expected {required_clusters} clusters, got {count}"
+        assert (
+            count == required_clusters
+        ), f"Expected {required_clusters} clusters, got {count}"
 
         # update the size
         entry.size_bytes = new_size
@@ -505,3 +571,49 @@ def write_file(self, entry: FatDirectoryEntry, data: bytes):
             cluster = self.next_cluster(cluster)
 
         assert len(data) == 0, "Data was not written completely, clusters missing"
+
+    def create_file(self, path: str):
+        """
+        Create a new file at the given path.
+        """
+        assert path[0] == "/", "Path must start with /"
+
+        path = path[1:]  # remove the leading /
+
+        parts = path.split("/")
+
+        directory_cluster = None
+        directory = self.read_root_directory()
+
+        parts, filename = parts[:-1], parts[-1]
+
+        for i, part in enumerate(parts):
+            current_entry = None
+            for entry in directory:
+                if entry.whole_name() == part:
+                    current_entry = entry
+                    break
+            if current_entry is None:
+                return None
+
+            if current_entry.attributes & 0x10 == 0:
+                raise Exception(f"{current_entry.whole_name()} is not a directory")
+            else:
+                directory = self.read_directory(current_entry.cluster)
+                directory_cluster = current_entry.cluster
+
+        # add new entry to the directory
+
+        filename, ext = filename.split(".")
+
+        if len(ext) > 3:
+            raise Exception("Ext must be 3 characters or less")
+        if len(filename) > 8:
+            raise Exception("Name must be 8 characters or less")
+
+        for c in filename + ext:
+
+            if c not in ALLOWED_FILE_CHARS:
+                raise Exception("Invalid character in filename")
+
+        return self.add_direntry(directory_cluster, filename, ext, 0)
tests/qemu-iotests/tests/vvfat +24 -5
index e0e23d1ab8..d8d802589d 100755
@@ -323,7 +323,7 @@ class TestVVFatDriver(QMPTestCase):
 
         with open(os.path.join(filesystem, "file0.txt"), "rb") as f:
             self.assertEqual(f.read(), new_content)
-    
+
     def test_write_large_file(self):
         """
         Test writing a large file
@@ -342,7 +342,7 @@ class TestVVFatDriver(QMPTestCase):
 
         with open(os.path.join(filesystem, "large1.txt"), "rb") as f:
             self.assertEqual(f.read(), new_content)
-    
+
     def test_truncate_file_change_clusters_less(self):
         """
         Test truncating a file by reducing the number of clusters
@@ -359,7 +359,6 @@ class TestVVFatDriver(QMPTestCase):
         with open(os.path.join(filesystem, "large1.txt"), "rb") as f:
             self.assertEqual(f.read(), b"A")
 
-    
     def test_write_file_change_clusters_less(self):
         """
         Test truncating a file by reducing the number of clusters
@@ -376,7 +375,7 @@ class TestVVFatDriver(QMPTestCase):
 
         with open(os.path.join(filesystem, "large2.txt"), "rb") as f:
             self.assertEqual(f.read(), new_content)
-    
+
     def test_write_file_change_clusters_more(self):
         """
         Test truncating a file by increasing the number of clusters
@@ -392,7 +391,27 @@ class TestVVFatDriver(QMPTestCase):
 
         with open(os.path.join(filesystem, "large2.txt"), "rb") as f:
             self.assertEqual(f.read(), new_content)
-    
+
+    def test_create_file(self):
+        """
+        Test creating a new file
+        """
+        fat16 = self.init_fat16()
+
+        new_file = fat16.create_file("/NEWFILE.TXT")
+
+        self.assertIsNotNone(new_file)
+        self.assertEqual(new_file.size_bytes, 0)
+
+        new_content = b"Hello, world! New file\n"
+        fat16.write_file(new_file, new_content)
+
+        self.assertEqual(fat16.read_file(new_file), new_content)
+
+        with open(os.path.join(filesystem, "newfile.txt"), "rb") as f:
+            self.assertEqual(f.read(), new_content)
+
+    # TODO: support deleting files
 
 
 if __name__ == "__main__":
tests/qemu-iotests/tests/vvfat.out +2 -2
index fa16b5ccef..6323079e08 100755
@@ -1,5 +1,5 @@
-.............
+..............
 ----------------------------------------------------------------------
-Ran 13 tests
+Ran 14 tests
 
 OK
Signature · 1 line
2.45.0
#9 lore raw
In reply to Amjad Alsharafi

Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:

Quoted 13 lines

These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.

Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash on abort() when the file is modified and a new cluster was added.

Quoted 4 lines

Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.

I was wondering how to reproduce the bugs that patches 2 and 3 fix. So I tried to run your iotests case, and while it does catch the bug that patch 1 fixes, it passes even without the other two fixes.

Is this expected? If so, can we add more tests that trigger the problems the other two patches address?

Kevin

#10 lore raw
Quoted 19 lines · Kevin Wolf

Am 26.05.2024 um 11:56 hat Amjad Alsharafi geschrieben:

Quoted 13 lines

These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.

Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash on abort() when the file is modified and a new cluster was added.

Quoted 4 lines

Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.

Quoted 9 lines

I was wondering how to reproduce the bugs that patches 2 and 3 fix. So I tried to run your iotests case, and while it does catch the bug that patch 1 fixes, it passes even without the other two fixes.

Is this expected? If so, can we add more tests that trigger the problems the other two patches address?

Kevin

Thanks for checking, so this bug happens when you have mapping for file, and the clusters are not contiguous.

For example, a file with clusters 12, 13, 15, here when trying to read from cluster 15, it will get the offset in the file by using the formula cluster_size * (15-12) (12 is the first cluster).

This is of course is not correct, and will result in error reading the file from outside the range.

The reason it wasn't clear when you tested it, is that since I'm modifying large2.txt, and its the last file in the disk, when trying to allocate new clusters, coincidentally, the new clusters are allocated after the last cluster of that same file, so the issue wasn't triggered.

I'll modify the test to use the other file, so that we can trigger the issue.

I'll also modify the other suggestions you had in the other patches and submit a new version.

Amjad