Skip to contentSkip to conversation

vvfat: Fix write bugs for large files and add iotests

PATCH Applied Amjad Alsharafi 2 participants 9 messages Jul 20, 2024 – Aug 9, 2024 Block layer

lore
Share & tools
lore
#0 lore raw

v6 cover vvfat: Fix write bugs for large files and add iotests

These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.

Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash on abort() when the file is modified and a new cluster was added.

Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.

v6:
- small fixes and code improvements when switching between active vfat mappings
- small fixes and code improvements for the fa16.py driver used for iotests

v5:
- Fix a bug in reading non-contiguous clusters in vvfat for more than 2 mappings.
- Added a test for adding more clusters where they are non-contiguous and result in 3 mappings (for the above fix).
- Split PATCH 2/4 into 2 patches and a better fix for the abort issue (now in PATCH 3/5).
- Other fixes and improvements in fat16.py module used for iotests.

v4:
Applied some suggestions from Kevin Wolf <[email protected]>:
- Fixed code formatting by following the coding style in scripts/checkpatch.pl
- Reduced changes related to iotests by setting vvfat format as non-generic.
- Added another test to cover the fix done in PATCH 2/4 and PATCH 3/4 for handling reading/writing files with non-continuous clusters.

v3:
Added test for creating new files in vvfat.

v2:
Added iotests for vvfat driver along with a simple fat16 module to run the tests.

v1:
https://patchew.org/QEMU/[email protected]/
Fix the issue of writing to the middle of the file in vvfat

Amjad Alsharafi (5):
vvfat: Fix bug in writing to middle of file
vvfat: Fix usage of info.file.offset
vvfat: Fix wrong checks for cluster mappings invariant
vvfat: Fix reading files with non-continuous clusters
iotests: Add vvfat tests

 block/vvfat.c                      |  34 +-
 tests/qemu-iotests/check           |   2 +-
 tests/qemu-iotests/fat16.py        | 675 +++++++++++++++++++++++++++++
 tests/qemu-iotests/testenv.py      |   2 +-
 tests/qemu-iotests/tests/vvfat     | 458 +++++++++++++++++++
 tests/qemu-iotests/tests/vvfat.out |   5 +
 6 files changed, 1157 insertions(+), 19 deletions(-)
 create mode 100644 tests/qemu-iotests/fat16.py
 create mode 100755 tests/qemu-iotests/tests/vvfat
 create mode 100755 tests/qemu-iotests/tests/vvfat.out
Signature · 1 line
2.45.2
#1 lore raw

PATCH v6 1/5 vvfat: Fix bug in writing to middle of file

Before this commit, the behavior when calling commit_one_file for example with offset=0x2000 (second cluster), what will happen is that we won't fetch the next cluster from the fat, and instead use the first cluster for the read operation.

This is due to off-by-one error here, where i=0x2000 !< offset=0x2000, thus not fetching the next cluster.

---
 block/vvfat.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
block/vvfat.c +2 -1
index 9d050ba3ae..19da009a5b 100644
@@ -2525,8 +2525,9 @@ commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset)
         return -1;
     }
 
-    for (i = s->cluster_size; i < offset; i += s->cluster_size)
+    for (i = 0; i < offset; i += s->cluster_size) {
         c = modified_fat_get(s, c);
+    }
 
     fd = qemu_open_old(mapping->path, O_RDWR | O_CREAT | O_BINARY, 0666);
     if (fd < 0) {
Signature · 1 line
2.45.2
#2 lore raw

PATCH v6 2/5 vvfat: Fix usage of `info.file.offset`

In reply to Amjad Alsharafi

The field is marked as "the offset in the file (in clusters)", but it was being used like this cluster_size*(nums)+mapping->info.file.offset, which is incorrect.

---
 block/vvfat.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
block/vvfat.c +7 -4
index 19da009a5b..247b232608 100644
@@ -1408,7 +1408,9 @@ read_cluster_directory:
 
         assert(s->current_fd);
 
-        offset=s->cluster_size*(cluster_num-s->current_mapping->begin)+s->current_mapping->info.file.offset;
+        offset = s->cluster_size *
+            ((cluster_num - s->current_mapping->begin)
+            + s->current_mapping->info.file.offset);
         if(lseek(s->current_fd, offset, SEEK_SET)!=offset)
             return -3;
         s->cluster=s->cluster_buffer;
@@ -1929,8 +1931,9 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const ch
                         (mapping->mode & MODE_DIRECTORY) == 0) {
 
                     /* was modified in qcow */
-                    if (offset != mapping->info.file.offset + s->cluster_size
-                            * (cluster_num - mapping->begin)) {
+                    if (offset != s->cluster_size
+                            * ((cluster_num - mapping->begin)
+                            + mapping->info.file.offset)) {
                         /* offset of this cluster in file chain has changed */
                         abort();
                         copy_it = 1;
@@ -2404,7 +2407,7 @@ static int commit_mappings(BDRVVVFATState* s,
                         (mapping->end - mapping->begin);
             } else
                 next_mapping->info.file.offset = mapping->info.file.offset +
-                        mapping->end - mapping->begin;
+                        (mapping->end - mapping->begin);
 
             mapping = next_mapping;
         }
Signature · 1 line
2.45.2
#3 lore raw

PATCH v6 3/5 vvfat: Fix wrong checks for cluster mappings invariant

In reply to Amjad Alsharafi

How this abort was intended to check for was:
- if the mapping->first_mapping_index is not the same as first_mapping_index, which **should** happen only in one case, when we are handling the first mapping, in that case mapping->first_mapping_index == -1, in all other cases, the other mappings after the first should have the condition true.
- From above, we know that this is the first mapping, so if the offset is not 0, then abort, since this is an invalid state.

The issue was that first_mapping_index is not set if we are checking from the middle, the variable first_mapping_index is only set if we passed through the check cluster_was_modified with the first mapping, and in the same function call we checked the other mappings.

One approach is to go into the loop even if cluster_was_modified is not true so that we will be able to set first_mapping_index for the first mapping, but since first_mapping_index is only used here, another approach is to just check manually for the mapping->first_mapping_index != -1 since we know that this is the value for the only entry where offset == 0 (i.e. first mapping).

---
 block/vvfat.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
block/vvfat.c +2 -8
index 247b232608..b63ac5d045 100644
@@ -1880,7 +1880,6 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const ch
 
     uint32_t cluster_num = begin_of_direntry(direntry);
     uint32_t offset = 0;
-    int first_mapping_index = -1;
     mapping_t* mapping = NULL;
     const char* basename2 = NULL;
 
@@ -1942,14 +1941,9 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const ch
 
                         if (strcmp(basename, basename2))
                             copy_it = 1;
-                        first_mapping_index = array_index(&(s->mapping), mapping);
-                    }
-
-                    if (mapping->first_mapping_index != first_mapping_index
-                            && mapping->info.file.offset > 0) {
-                        abort();
-                        copy_it = 1;
                     }
+                    assert(mapping->first_mapping_index == -1
+                            || mapping->info.file.offset > 0);
 
                     /* need to write out? */
                     if (!was_modified && is_file(direntry)) {
Signature · 1 line
2.45.2
#4 lore raw

PATCH v6 4/5 vvfat: Fix reading files with non-continuous clusters

In reply to Amjad Alsharafi

When reading with read_cluster we get the mapping with find_mapping_for_cluster and then we call open_file for this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters 500 -> 503, this will give us 2 mappings one has the range 500..501 and another 503..504, both point to the same file, but different offsets.

When we don't open the file since the path is the same, we won't assign s->current_mapping and thus accessing way out of bound of the file.

From our example above, after open_file (that didn't open anything) we will get the offset into the file with s->cluster_size*(cluster_num-s->current_mapping->begin), which will give us 0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.

---
 block/vvfat.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
block/vvfat.c +6 -4
index b63ac5d045..d2b879705e 100644
@@ -1360,15 +1360,17 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping)
 {
     if(!mapping)
         return -1;
-    if(!s->current_mapping ||
-            strcmp(s->current_mapping->path,mapping->path)) {
+    if (s->current_mapping != mapping) {
         /* open file */
         int fd = qemu_open_old(mapping->path,
-                               O_RDONLY | O_BINARY | O_LARGEFILE);
-        if(fd<0)
+                            O_RDONLY | O_BINARY | O_LARGEFILE);
+        if (fd < 0) {
             return -1;
+        }
         vvfat_close_current_file(s);
+
         s->current_fd = fd;
+        assert(s->current_fd);
         s->current_mapping = mapping;
     }
     return 0;
Signature · 1 line
2.45.2
#5 lore raw

Re: [PATCH v6 4/5] vvfat: Fix reading files with non-continuous clusters

Am 20.07.2024 um 12:13 hat Amjad Alsharafi geschrieben:

Quoted 38 lines

When reading with read_cluster we get the mapping with find_mapping_for_cluster and then we call open_file for this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters 500 -> 503, this will give us 2 mappings one has the range 500..501 and another 503..504, both point to the same file, but different offsets.

When we don't open the file since the path is the same, we won't assign s->current_mapping and thus accessing way out of bound of the file.

From our example above, after open_file (that didn't open anything) we will get the offset into the file with s->cluster_size*(cluster_num-s->current_mapping->begin), which will give us 0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.

---
 block/vvfat.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
block/vvfat.c +2 -4
index b63ac5d045..d2b879705e 100644
@@ -1360,15 +1360,17 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping)
 {
     if(!mapping)
         return -1;
-    if(!s->current_mapping ||
-            strcmp(s->current_mapping->path,mapping->path)) {
+    if (s->current_mapping != mapping) {
         /* open file */
         int fd = qemu_open_old(mapping->path,
-                               O_RDONLY | O_BINARY | O_LARGEFILE);
-        if(fd<0)
+                            O_RDONLY | O_BINARY | O_LARGEFILE);

block/vvfat.c static int open_file(BDRVVVFATState* s,mapping_t* mapping) · line 1376

Quoted 10 lines
block/vvfat.c +4 -0
+        if (fd < 0) {
             return -1;
+        }
         vvfat_close_current_file(s);
+
         s->current_fd = fd;
+        assert(s->current_fd);
         s->current_mapping = mapping;
     }
     return 0;

Now we're reopening the file even if the mapping is actually referring to the same file that was already open. So the result should be correct, but wasteful in what is probably a common case.

However, this version of the patch simplified things enough that I think I finally see what we really want:

block/vvfat.c +2 -1
index e3a83fbc88..8ffe8b3b9b 100644
@@ -1369,8 +1369,9 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping)
             return -1;
         vvfat_close_current_file(s);
         s->current_fd = fd;
-        s->current_mapping = mapping;
     }
+
+    s->current_mapping = mapping;
     return 0;
 }
 

That should be all that is needed, and it passes your test case.

I don't usually do this, but since time is running out for QEMU 9.1, I'll just replace the code of this patch with the above and go ahead with that. If you think it's wrong, let me know and we'll fix it on top of this series.

Kevin

#6 lore raw

Re: [PATCH v6 4/5] vvfat: Fix reading files with non-continuous clusters

Quoted 63 lines · Kevin Wolf

Am 20.07.2024 um 12:13 hat Amjad Alsharafi geschrieben:

Quoted 48 lines

When reading with read_cluster we get the mapping with find_mapping_for_cluster and then we call open_file for this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters 500 -> 503, this will give us 2 mappings one has the range 500..501 and another 503..504, both point to the same file, but different offsets.

When we don't open the file since the path is the same, we won't assign s->current_mapping and thus accessing way out of bound of the file.

From our example above, after open_file (that didn't open anything) we will get the offset into the file with s->cluster_size*(cluster_num-s->current_mapping->begin), which will give us 0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.

---
 block/vvfat.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
block/vvfat.c +6 -4
index b63ac5d045..d2b879705e 100644
@@ -1360,15 +1360,17 @@ static int open_file(BDRVVVFATState*
 s,mapping_t* mapping)
 {
     if(!mapping)
         return -1;
-    if(!s->current_mapping ||
-            strcmp(s->current_mapping->path,mapping->path)) {
+    if (s->current_mapping != mapping) {
         /* open file */
         int fd = qemu_open_old(mapping->path,
-                               O_RDONLY | O_BINARY | O_LARGEFILE);
-        if(fd<0)
+                            O_RDONLY | O_BINARY | O_LARGEFILE);
+        if (fd < 0) {
             return -1;
+        }
         vvfat_close_current_file(s);
+
         s->current_fd = fd;
+        assert(s->current_fd);
         s->current_mapping = mapping;
     }
Quoted 1 line
     return 0;

Now we're reopening the file even if the mapping is actually referring to the same file that was already open. So the result should be correct, but wasteful in what is probably a common case.

However, this version of the patch simplified things enough that I think I finally see what we really want:

block/vvfat.c +0 -0
index e3a83fbc88..8ffe8b3b9b 100644

block/vvfat.c static int open_file(BDRVVVFATState* · line 1377

Quoted 10 lines
block/vvfat.c +2 -1
@@ -1369,8 +1369,9 @@ static int open_file(BDRVVVFATState*
 s,mapping_t* mapping)
            return -1;
        vvfat_close_current_file(s);
        s->current_fd = fd;
-        s->current_mapping = mapping;
    }
+
+    s->current_mapping = mapping;
    return 0;
 }

Ah, that indeed fixes it with such a simple change haha.

Quoted 11 lines

That should be all that is needed, and it passes your test case.

I don't usually do this, but since time is running out for QEMU 9.1, I'll just replace the code of this patch with the above and go ahead with that. If you think it's wrong, let me know and we'll fix it on top of this series.

Kevin

Thanks for that, and for publishing this. Looks good.

Amjad

#7 lore raw

PATCH v6 5/5 iotests: Add `vvfat` tests

In reply to Amjad Alsharafi

Added several tests to verify the implementation of the vvfat driver.

We needed a way to interact with it, so created a basic fat16.py driver that handled writing correct sectors for us.

Added vvfat to the non-generic formats, as its not a normal image format.

---
 tests/qemu-iotests/check           |   2 +-
 tests/qemu-iotests/fat16.py        | 675 +++++++++++++++++++++++++++++
 tests/qemu-iotests/testenv.py      |   2 +-
 tests/qemu-iotests/tests/vvfat     | 458 +++++++++++++++++++
 tests/qemu-iotests/tests/vvfat.out |   5 +
 5 files changed, 1140 insertions(+), 2 deletions(-)
 create mode 100644 tests/qemu-iotests/fat16.py
 create mode 100755 tests/qemu-iotests/tests/vvfat
 create mode 100755 tests/qemu-iotests/tests/vvfat.out
tests/qemu-iotests/check +1 -1
index 56d88ca423..545f9ec7bd 100755
@@ -84,7 +84,7 @@ def make_argparser() -> argparse.ArgumentParser:
     p.set_defaults(imgfmt='raw', imgproto='file')
 
     format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2',
-                   'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg']
+                   'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg', 'vvfat']
     g_fmt = p.add_argument_group(
         '  image format options',
         'The following options set the IMGFMT environment variable. '
tests/qemu-iotests/fat16.py new file +675 -0

675 lines not on this page · raw message

tests/qemu-iotests/testenv.py +1 -1
index 588f30a4f1..4053d29de4 100644
@@ -250,7 +250,7 @@ def __init__(self, source_dir: str, build_dir: str,
         self.qemu_img_options = os.getenv('QEMU_IMG_OPTIONS')
         self.qemu_nbd_options = os.getenv('QEMU_NBD_OPTIONS')
 
-        is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg']
+        is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg', 'vvfat']
         self.imgfmt_generic = 'true' if is_generic else 'false'
 
         self.qemu_io_options = f'--cache {self.cachemode} --aio {self.aiomode}'
tests/qemu-iotests/tests/vvfat new file +458 -0

458 lines not on this page · raw message

tests/qemu-iotests/tests/vvfat.out new file +5 -0
new file mode 100755
index 0000000000..b6f257674e
@@ -0,0 +1,5 @@
+................
+----------------------------------------------------------------------
+Ran 16 tests
+
+OK
Signature · 1 line
2.45.2
Applied
#8 lore raw
In reply to Amjad Alsharafi

Am 20.07.2024 um 12:13 hat Amjad Alsharafi geschrieben:

Quoted 13 lines

These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.

Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash on abort() when the file is modified and a new cluster was added.

Quoted 4 lines

Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.

Thanks, applied to the block branch (with patch 4 changed as indicated in my comment there).

Kevin