vvfat: Fix write bugs for large files and add iotests
PATCH v6 1/5 vvfat: Fix bug in writing to middle of file
In reply to Amjad AlsharafiBefore this commit, the behavior when calling commit_one_file for example with offset=0x2000 (second cluster), what will happen is that we won't fetch the next cluster from the fat, and instead use the first cluster for the read operation.
This is due to off-by-one error here, where i=0x2000 !< offset=0x2000, thus not fetching the next cluster.
--- block/vvfat.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
block/vvfat.c +2 -1
@@ -2525,8 +2525,9 @@ commit_one_file(BDRVVVFATState* s, int dir_index, uint32_t offset)return -1 ; } -for (i = s->cluster_size; i < offset; i += s->cluster_size) +for (i =0 ; i < offset; i += s->cluster_size) { c =modified_fat_get (s, c); + } fd =qemu_open_old (mapping->path,O_RDWR |O_CREAT |O_BINARY ,0666 );if (fd <0 ) {
Signature · 1 line
2.45.2
PATCH v6 2/5 vvfat: Fix usage of `info.file.offset`
In reply to Amjad AlsharafiThe field is marked as "the offset in the file (in clusters)", but it was being used like this cluster_size*(nums)+mapping->info.file.offset, which is incorrect.
--- block/vvfat.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
block/vvfat.c +7 -4
@@ -1408,7 +1408,9 @@ read_cluster_directory:assert (s->current_fd); - offset=s->cluster_size*(cluster_num-s->current_mapping->begin)+s->current_mapping->info.file.offset; + offset = s->cluster_size * + ((cluster_num - s->current_mapping->begin) + + s->current_mapping->info.file.offset);if (lseek (s->current_fd, offset,SEEK_SET )!=offset)return -3 ; s->cluster=s->cluster_buffer;
@@ -1929,8 +1931,9 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const ch (mapping->mode & MODE_DIRECTORY) == 0) {/* was modified in qcow */ -if (offset != mapping->info.file.offset + s->cluster_size - * (cluster_num - mapping->begin)) { +if (offset != s->cluster_size + * ((cluster_num - mapping->begin) + + mapping->info.file.offset)) {/* offset of this cluster in file chain has changed */ abort (); copy_it =1 ;
@@ -2404,7 +2407,7 @@ static int commit_mappings(BDRVVVFATState* s, (mapping->end - mapping->begin); }else next_mapping->info.file.offset = mapping->info.file.offset + - mapping->end - mapping->begin; + (mapping->end - mapping->begin); mapping = next_mapping; }
Signature · 1 line
2.45.2
v6 cover vvfat: Fix write bugs for large files and add iotests
These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.
Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash on abort() when the file is modified and a new cluster was added.
Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.
v6:
- small fixes and code improvements when switching between active vfat mappings
- small fixes and code improvements for the fa16.py driver used for iotests
v5:
- Fix a bug in reading non-contiguous clusters in vvfat for more than 2 mappings.
- Added a test for adding more clusters where they are non-contiguous and result in 3 mappings (for the above fix).
- Split PATCH 2/4 into 2 patches and a better fix for the abort issue (now in PATCH 3/5).
- Other fixes and improvements in fat16.py module used for iotests.
v4:
Applied some suggestions from Kevin Wolf <[email protected]>:
- Fixed code formatting by following the coding style in scripts/checkpatch.pl
- Reduced changes related to iotests by setting vvfat format as non-generic.
- Added another test to cover the fix done in PATCH 2/4 and PATCH 3/4 for handling reading/writing files with non-continuous clusters.
v3:
Added test for creating new files in vvfat.
v2:
Added iotests for vvfat driver along with a simple fat16 module to run the tests.
v1:
https://patchew.org/QEMU/[email protected]/
Fix the issue of writing to the middle of the file in vvfat
Amjad Alsharafi (5):
vvfat: Fix bug in writing to middle of file
vvfat: Fix usage of info.file.offset
vvfat: Fix wrong checks for cluster mappings invariant
vvfat: Fix reading files with non-continuous clusters
iotests: Add vvfat tests
block/vvfat.c | 34 +- tests/qemu-iotests/check | 2 +- tests/qemu-iotests/fat16.py | 675 +++++++++++++++++++++++++++++ tests/qemu-iotests/testenv.py | 2 +- tests/qemu-iotests/tests/vvfat | 458 +++++++++++++++++++ tests/qemu-iotests/tests/vvfat.out | 5 + 6 files changed, 1157 insertions(+), 19 deletions(-) create mode 100644 tests/qemu-iotests/fat16.py create mode 100755 tests/qemu-iotests/tests/vvfat create mode 100755 tests/qemu-iotests/tests/vvfat.out
Signature · 1 line
2.45.2
PATCH v6 5/5 iotests: Add `vvfat` tests
Added several tests to verify the implementation of the vvfat driver.
We needed a way to interact with it, so created a basic fat16.py driver that handled writing correct sectors for us.
Added vvfat to the non-generic formats, as its not a normal image format.
--- tests/qemu-iotests/check | 2 +- tests/qemu-iotests/fat16.py | 675 +++++++++++++++++++++++++++++ tests/qemu-iotests/testenv.py | 2 +- tests/qemu-iotests/tests/vvfat | 458 +++++++++++++++++++ tests/qemu-iotests/tests/vvfat.out | 5 + 5 files changed, 1140 insertions(+), 2 deletions(-) create mode 100644 tests/qemu-iotests/fat16.py create mode 100755 tests/qemu-iotests/tests/vvfat create mode 100755 tests/qemu-iotests/tests/vvfat.out
tests/qemu-iotests/check +1 -1
@@ -84,7 +84,7 @@ def make_argparser() -> argparse.ArgumentParser: p.set_defaults(imgfmt='raw', imgproto='file') format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2', - 'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg'] + 'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg', 'vvfat'] g_fmt = p.add_argument_group( ' image format options', 'The following options set the IMGFMT environment variable. '
tests/qemu-iotests/fat16.py new file +675 -0
675 lines not on this page · raw message
tests/qemu-iotests/testenv.py +1 -1
@@ -250,7 +250,7 @@ def __init__(self, source_dir: str, build_dir: str, self.qemu_img_options = os.getenv('QEMU_IMG_OPTIONS') self.qemu_nbd_options = os.getenv('QEMU_NBD_OPTIONS') - is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg'] + is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg', 'vvfat'] self.imgfmt_generic = 'true' if is_generic else 'false' self.qemu_io_options = f'--cache {self.cachemode} --aio {self.aiomode}'
tests/qemu-iotests/tests/vvfat new file +458 -0
458 lines not on this page · raw message
tests/qemu-iotests/tests/vvfat.out new file +5 -0
@@ -0,0 +1,5 @@ +................ +---------------------------------------------------------------------- +Ran 16 tests + +OK
Signature · 1 line
2.45.2
PATCH v6 4/5 vvfat: Fix reading files with non-continuous clusters
In reply to Amjad AlsharafiWhen reading with read_cluster we get the mapping with find_mapping_for_cluster and then we call open_file for this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters 500 -> 503, this will give us 2 mappings one has the range 500..501 and another 503..504, both point to the same file, but different offsets.
When we don't open the file since the path is the same, we won't assign s->current_mapping and thus accessing way out of bound of the file.
From our example above, after open_file (that didn't open anything) we will get the offset into the file with s->cluster_size*(cluster_num-s->current_mapping->begin), which will give us 0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.
--- block/vvfat.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
block/vvfat.c +6 -4
@@ -1360,15 +1360,17 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping) {if (!mapping)return -1 ; -if (!s->current_mapping || -strcmp (s->current_mapping->path,mapping->path)) { +if (s->current_mapping != mapping) {/* open file */ int fd =qemu_open_old (mapping->path, -O_RDONLY |O_BINARY |O_LARGEFILE ); -if (fd<0 ) +O_RDONLY |O_BINARY |O_LARGEFILE ); +if (fd <0 ) {return -1 ; + }vvfat_close_current_file (s); + s->current_fd = fd; +assert (s->current_fd); s->current_mapping = mapping; }return 0 ;
Signature · 1 line
2.45.2
PATCH v6 3/5 vvfat: Fix wrong checks for cluster mappings invariant
In reply to Amjad AlsharafiHow this abort was intended to check for was:
- if the mapping->first_mapping_index is not the same as first_mapping_index, which **should** happen only in one case, when we are handling the first mapping, in that case mapping->first_mapping_index == -1, in all other cases, the other mappings after the first should have the condition true.
- From above, we know that this is the first mapping, so if the offset is not 0, then abort, since this is an invalid state.
The issue was that first_mapping_index is not set if we are checking from the middle, the variable first_mapping_index is only set if we passed through the check cluster_was_modified with the first mapping, and in the same function call we checked the other mappings.
One approach is to go into the loop even if cluster_was_modified is not true so that we will be able to set first_mapping_index for the first mapping, but since first_mapping_index is only used here, another approach is to just check manually for the mapping->first_mapping_index != -1 since we know that this is the value for the only entry where offset == 0 (i.e. first mapping).
--- block/vvfat.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
block/vvfat.c +2 -8
@@ -1880,7 +1880,6 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const chuint32_t cluster_num =begin_of_direntry (direntry);uint32_t offset =0 ; -int first_mapping_index =-1 ;mapping_t * mapping =NULL ;const char * basename2 =NULL ;
@@ -1942,14 +1941,9 @@ get_cluster_count_for_direntry(BDRVVVFATState* s, direntry_t* direntry, const chif (strcmp (basename, basename2)) copy_it =1 ; - first_mapping_index =array_index (&(s->mapping), mapping); - } - -if (mapping->first_mapping_index != first_mapping_index - && mapping->info.file.offset >0 ) { -abort (); - copy_it =1 ; } +assert (mapping->first_mapping_index ==-1 + || mapping->info.file.offset >0 );/* need to write out? */ if (!was_modified &&is_file (direntry)) {
Signature · 1 line
2.45.2
Re: [PATCH v6 4/5] vvfat: Fix reading files with non-continuous clusters
In reply to Amjad AlsharafiAm 20.07.2024 um 12:13 hat Amjad Alsharafi geschrieben:
Quoted 38 lines
When reading with
read_clusterwe get themappingwithfind_mapping_for_clusterand then we callopen_filefor this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters500 -> 503, this will give us 2 mappings one has the range500..501and another503..504, both point to the same file, but different offsets.When we don't open the file since the path is the same, we won't assign
s->current_mappingand thus accessing way out of bound of the file.From our example above, after
open_file(that didn't open anything) we will get the offset into the file withs->cluster_size*(cluster_num-s->current_mapping->begin), which will give us0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.Signed-off-by Amjad Alsharafi <[email protected]>--- block/vvfat.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)block/vvfat.c +2
-4@@ -1360,15 +1360,17 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping) {if (!mapping)return -1 ; -if (!s->current_mapping || -strcmp (s->current_mapping->path,mapping->path)) { +if (s->current_mapping != mapping) {/* open file */ int fd =qemu_open_old (mapping->path, -O_RDONLY |O_BINARY |O_LARGEFILE ); -if (fd<0 ) +O_RDONLY |O_BINARY |O_LARGEFILE );
block/vvfat.c static int open_file(BDRVVVFATState* s,mapping_t* mapping) · line 1376
Quoted 10 lines
block/vvfat.c +4
-0+if (fd <0 ) {return -1 ; + }vvfat_close_current_file (s); + s->current_fd = fd; +assert (s->current_fd); s->current_mapping = mapping; }return 0 ;
Now we're reopening the file even if the mapping is actually referring to the same file that was already open. So the result should be correct, but wasteful in what is probably a common case.
However, this version of the patch simplified things enough that I think I finally see what we really want:
block/vvfat.c +2 -1
@@ -1369,8 +1369,9 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping)return -1 ;vvfat_close_current_file (s); s->current_fd = fd; - s->current_mapping = mapping; } + + s->current_mapping = mapping;return 0 ; }
That should be all that is needed, and it passes your test case.
I don't usually do this, but since time is running out for QEMU 9.1, I'll just replace the code of this patch with the above and go ahead with that. If you think it's wrong, let me know and we'll fix it on top of this series.
Kevin
Kevin Wolf
AppliedAm 20.07.2024 um 12:13 hat Amjad Alsharafi geschrieben:
Quoted 13 lines
These patches fix some bugs found when modifying files in vvfat. First, there was a bug when writing to the cluster 2 or above of a file, it will copy the cluster before it instead, so, when writing to cluster=2, the content of cluster=1 will be copied into disk instead in its place.
Another issue was modifying the clusters of a file and adding new clusters, this showed 2 issues:
- If the new cluster is not immediately after the last cluster, it will cause issues when reading from this file in the future.
- Generally, the usage of info.file.offset was incorrect, and the system would crash onabort()when the file is modified and a new cluster was added.
Quoted 4 lines
Also, added some iotests for vvfat, covering the this fix and also general behavior such as reading, writing, and creating files on the filesystem. Including tests for reading/writing the first cluster which
would pass even before this patch.
Thanks, applied to the block branch (with patch 4 changed as indicated in my comment there).
Kevin
Re: [PATCH v6 4/5] vvfat: Fix reading files with non-continuous clusters
In reply to Kevin WolfQuoted 63 lines · Kevin Wolf
Am 20.07.2024 um 12:13 hat Amjad Alsharafi geschrieben:
Quoted 48 lines
When reading with
read_clusterwe get themappingwithfind_mapping_for_clusterand then we callopen_filefor this mapping.
The issue appear when its the same file, but a second cluster that is not immediately after it, imagine clusters500 -> 503, this will give us 2 mappings one has the range500..501and another503..504, both point to the same file, but different offsets.When we don't open the file since the path is the same, we won't assign
s->current_mappingand thus accessing way out of bound of the file.From our example above, after
open_file(that didn't open anything) we will get the offset into the file withs->cluster_size*(cluster_num-s->current_mapping->begin), which will give us0x2000 * (504-500), which is out of bound for this mapping and will produce some issues.Signed-off-by Amjad Alsharafi <[email protected]>--- block/vvfat.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)block/vvfat.c +6
-4@@ -1360,15 +1360,17 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping) {if (!mapping)return -1 ; -if (!s->current_mapping || -strcmp (s->current_mapping->path,mapping->path)) { +if (s->current_mapping != mapping) {/* open file */ int fd =qemu_open_old (mapping->path, -O_RDONLY |O_BINARY |O_LARGEFILE ); -if (fd<0 ) +O_RDONLY |O_BINARY |O_LARGEFILE ); +if (fd <0 ) {return -1 ; + }vvfat_close_current_file (s); + s->current_fd = fd; +assert (s->current_fd); s->current_mapping = mapping; }Quoted 1 line
return 0;Now we're reopening the file even if the mapping is actually referring to the same file that was already open. So the result should be correct, but wasteful in what is probably a common case.
However, this version of the patch simplified things enough that I think I finally see what we really want:
block/vvfat.c +0
-0
block/vvfat.c static int open_file(BDRVVVFATState* · line 1377
Quoted 10 lines
block/vvfat.c +2
-1@@ -1369,8 +1369,9 @@ static int open_file(BDRVVVFATState* s,mapping_t* mapping)return -1 ;vvfat_close_current_file (s); s->current_fd = fd; - s->current_mapping = mapping; } + + s->current_mapping = mapping;return 0 ; }
Ah, that indeed fixes it with such a simple change haha.
Quoted 11 lines
That should be all that is needed, and it passes your test case.
I don't usually do this, but since time is running out for QEMU 9.1, I'll just replace the code of this patch with the above and go ahead with that. If you think it's wrong, let me know and we'll fix it on top of this series.
Kevin
Thanks for that, and for publishing this. Looks good.
Amjad