block/file-posix: Fix fully preallocated truncate
cover block/file-posix: Fix fully preallocated truncate
Fully preallocated truncation has a 50 % chance of working on images files over file-posix. It works if $SIZE % 4G < 2G, and it fails otherwise. To make things even more interesting, often you would not even notice because qemu reported success even though it did nothing (because after the successful lseek(), errno was still 0, so when the file-posix driver tried to return a negative error code, it actually reported success).
This issue is fixed by patch 1 in this series. Thanks to Daniel for reporting!
Max Reitz (2):
block/file-posix: Fix fully preallocated truncate
iotests: Test preallocated truncate of 2G image
block/file-posix.c | 5 +++-- tests/qemu-iotests/106 | 24 ++++++++++++++++++++++++ tests/qemu-iotests/106.out | 10 ++++++++++ 3 files changed, 37 insertions(+), 2 deletions(-)
Signature · 1 line
2.14.3
PATCH 1/2 block/file-posix: Fix fully preallocated truncate
Storing the lseek() result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine.
So we should use the correct type for storing the result: off_t.
--- block/file-posix.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
block/file-posix.c +3 -2
@@ -1697,6 +1697,7 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc,case PREALLOC_MODE_FULL : {int64_t num =0 , left = offset - current_length; +off_t seek_result;/* * Knowing the final size from the beginning could allow the file
@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) { result = -errno;error_setg_errno (errp, -result,"Failed to seek to the old end of file" );
Signature · 1 line
2.14.3
Quoted 28 lines · Max Reitz
Storing the
lseek()result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine.So we should use the correct type for storing the result:
off_t.Reported-by Daniel P. Berrange <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/file-posix.c +1
-0@@ -1697,6 +1697,7 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc,case PREALLOC_MODE_FULL : {int64_t num =0 , left = offset - current_length; +off_t seek_result;/* * Knowing the final size from the beginning could allow the file
block/file-posix.c static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, · line 1716
Quoted 7 lines
block/file-posix.c +2
-2@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) {
off_t is an unsigned type, so this comparison to "< 0" is bogus - only the exact value (off_t)-1 indicates an error. So this needs to be
if (seek_result == (off_t)-1) {
...
}Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 40 lines · Daniel P. Berrangé
Quoted 28 lines · Max Reitz
Storing the
lseek()result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine.So we should use the correct type for storing the result:
off_t.Reported-by Daniel P. Berrange <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/file-posix.c +1
-0@@ -1697,6 +1697,7 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc,case PREALLOC_MODE_FULL : {int64_t num =0 , left = offset - current_length; +off_t seek_result;/* * Knowing the final size from the beginning could allow the file block/file-posix.c static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, · line 1716
Quoted 7 lines
block/file-posix.c +2
-2@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) {
off_tis an unsigned type, so this comparison to "< 0" is bogus - only the exact value (off_t)-1 indicates an error. So this needs to be
Quoted 3 lines
if (seek_result == (off_t)-1) {
...
}Hmmm... On my system, it appears to be a long int[1]. And find_allocation() does an off_t < 0 comparison already. And "man 0p sys_types.h" says "blkcnt_t and off_t shall be signed integer types."
Max
[1]
#define do_stringify(x) #x
#define stringify(x) do_stringify(x)
int main(void)
{
printf("%s\n", stringify(__OFF_T_TYPE));
}Output: long int
Quoted 41 lines · Max Reitz
Quoted 40 lines · Daniel P. Berrangé
Quoted 28 lines · Max Reitz
Storing the
lseek()result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine.So we should use the correct type for storing the result:
off_t.Reported-by Daniel P. Berrange <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/file-posix.c +1
-0@@ -1697,6 +1697,7 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc,case PREALLOC_MODE_FULL : {int64_t num =0 , left = offset - current_length; +off_t seek_result;/* * Knowing the final size from the beginning could allow the file block/file-posix.c static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, · line 1716
Quoted 7 lines
block/file-posix.c +2
-2@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) {
off_tis an unsigned type, so this comparison to "< 0" is bogus - only the exact value (off_t)-1 indicates an error. So this needs to be
Quoted 8 lines
Quoted 3 lines
if (seek_result == (off_t)-1) { ... }Hmmm... On my system, it appears to be a long int[1]. And
find_allocation()does anoff_t< 0 comparison already. And "man 0psys_types.h" says "blkcnt_tandoff_tshall be signed integer types."
Hmm, that's odd then - lseek man page explicitly said it must be cast, which suggested to me it could be unsigned:
RETURN VALUE
Upon successful completion, lseek() returns the resulting offset loca‐ tion as measured in bytes from the beginning of the file. On error, the value (off_t) -1 is returned and errno is set to indicate the error.
CC'ing Eric for the "official" POSIX answer....
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 54 lines · Daniel P. Berrangé
Quoted 41 lines · Max Reitz
Quoted 40 lines · Daniel P. Berrangé
Quoted 28 lines · Max Reitz
Storing the
lseek()result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine.So we should use the correct type for storing the result:
off_t.Reported-by Daniel P. Berrange <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/file-posix.c +1
-0@@ -1697,6 +1697,7 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc,case PREALLOC_MODE_FULL : {int64_t num =0 , left = offset - current_length; +off_t seek_result;/* * Knowing the final size from the beginning could allow the file block/file-posix.c static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, · line 1716
Quoted 7 lines
block/file-posix.c +2
-2@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) {
off_tis an unsigned type, so this comparison to "< 0" is bogus - only the exact value (off_t)-1 indicates an error. So this needs to beQuoted 8 lines
Quoted 3 lines
if (seek_result == (off_t)-1) { ... }Hmmm... On my system, it appears to be a long int[1]. And
find_allocation()does anoff_t< 0 comparison already. And "man 0psys_types.h" says "blkcnt_tandoff_tshall be signed integer types."Hmm, that's odd then - lseek man page explicitly said it must be cast, which suggested to me it could be unsigned:
Quoted 7 lines
RETURN VALUE
Upon successful completion,lseek()returns the resulting offset loca‐ tion as measured in bytes from the beginning of the file. On error, the value (off_t) -1 is returned and errno is set to indicate the error.CC'ing Eric for the "official" POSIX answer....
But it also says (under NOTES):
The off_t data type is a signed integer data type specified by POSIX.1.
Max
Quoted 61 lines · Max Reitz
Quoted 60 lines · Daniel P. Berrangé
Quoted 41 lines · Max Reitz
Quoted 40 lines · Daniel P. Berrangé
Quoted 28 lines · Max Reitz
Storing the
lseek()result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine.So we should use the correct type for storing the result:
off_t.Reported-by Daniel P. Berrange <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/file-posix.c +1
-0@@ -1697,6 +1697,7 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc,case PREALLOC_MODE_FULL : {int64_t num =0 , left = offset - current_length; +off_t seek_result;/* * Knowing the final size from the beginning could allow the file block/file-posix.c static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, · line 1716
Quoted 7 lines
block/file-posix.c +2
-2@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) {
off_tis an unsigned type, so this comparison to "< 0" is bogus - only the exact value (off_t)-1 indicates an error. So this needs to beQuoted 8 lines
Quoted 3 lines
if (seek_result == (off_t)-1) { ... }Hmmm... On my system, it appears to be a long int[1]. And
find_allocation()does anoff_t< 0 comparison already. And "man 0psys_types.h" says "blkcnt_tandoff_tshall be signed integer types."Hmm, that's odd then - lseek man page explicitly said it must be cast, which suggested to me it could be unsigned:
RETURN VALUE
Upon successful completion,lseek()returns the resulting offset loca‐ tion as measured in bytes from the beginning of the file. On error, the value (off_t) -1 is returned and errno is set to indicate the error.
Quoted 5 lines
Quoted 1 line
CC'ing Eric for the "official" POSIX answer....
But it also says (under NOTES):
The
off_tdata type is a signed integer data type specified by POSIX.1.
Ok, lets ignore my comments then -the "< 0" vs "!= -1" difference is harmless given this.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 12 lines · Daniel P. Berrangé
Quoted 12 lines
Quoted 12 lines
block/file-posix.c static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, · line 1716
Quoted 8 lines
block/file-posix.c +2
-2@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) {
off_tis an unsigned type, so this comparison to "< 0" is bogus - only the exact value (off_t)-1 indicates an error. So this needs to be
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
if (seek_result == (off_t)-1) { ... }
No, off_t is ALWAYS signed, even when it is 32-bit. The cast helps code that is written for 64-bit off_t to still work when compiled with the small file model where a 32-bit value is used (but we don't have to worry about that, as we always compile for large file mode with 64-bit off_t).
Quoted 9 lines
Quoted 5 lines
Hmmm... On my system, it appears to be a long int[1]. And
find_allocation()does anoff_t< 0 comparison already. And "man 0psys_types.h" says "blkcnt_tandoff_tshall be signed integer types."Hmm, that's odd then - lseek man page explicitly said it must be cast, which suggested to me it could be unsigned:
Quoted 7 lines
RETURN VALUE
Upon successful completion,lseek()returns the resulting offset loca‐ tion as measured in bytes from the beginning of the file. On error, the value (off_t) -1 is returned and errno is set to indicate the error.CC'ing Eric for the "official" POSIX answer....
It MAY be that the man page mentions a cast because '-1' is an int but '(off_t) -1' can be larger than an int. But it may also be that you've uncovered something worth reporting as a bug to the man page project.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Quoted 28 lines · Max Reitz
Storing the
lseek()result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine.So we should use the correct type for storing the result:
off_t.Reported-by Daniel P. Berrange <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/file-posix.c +1
-0@@ -1697,6 +1697,7 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc,case PREALLOC_MODE_FULL : {int64_t num =0 , left = offset - current_length; +off_t seek_result;/* * Knowing the final size from the beginning could allow the file
block/file-posix.c static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, · line 1719
Quoted 10 lines
block/file-posix.c +2
-2@@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf =g_malloc0 (65536 ); - result =lseek (fd, current_length,SEEK_SET ); -if (result <0 ) { + seek_result =lseek (fd, current_length,SEEK_SET ); +if (seek_result <0 ) { result = -errno;error_setg_errno (errp, -result,"Failed to seek to the old end of file" );
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 2/2 iotests: Test preallocated truncate of 2G image
In reply to Max Reitz--- tests/qemu-iotests/106 | 24 ++++++++++++++++++++++++ tests/qemu-iotests/106.out | 10 ++++++++++ 2 files changed, 34 insertions(+)
tests/qemu-iotests/106 +24 -0
@@ -86,6 +86,30 @@ for growth_mode in falloc full off; do $QEMU_IMG resize -f "$IMGFMT" --shrink --preallocation=$growth_mode "$TEST_IMG" -${GROWTH_SIZE}K done +echo +echo '=== Testing image growth on 2G empty image ===' + +for growth_mode in falloc full; do + echo + echo "--- growth_mode=$growth_mode ---" + + # Maybe we want to do an lseek() to the end of the file before the + # preallocation; if the file has a length of 2 GB, that would + # return an integer that overflows to negative when put into a + # plain int. We should use the correct type for the result, and + # this tests we do. + + _make_test_img 2G + $QEMU_IMG resize -f "$IMGFMT" --preallocation=$growth_mode "$TEST_IMG" +${GROWTH_SIZE}K + + actual_size=$($QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep 'disk size') + actual_size=$(echo "$actual_size" | sed -e 's/^[^0-9]*\([0-9]\+\).*$/\1/') + + if [ $actual_size -lt $GROWTH_SIZE ]; then + echo "ERROR: Image should have at least ${GROWTH_SIZE}K, but has ${actual_size}K" + fi +done + # success, all done echo '*** done' rm -f $seq.full
tests/qemu-iotests/106.out +10 -0
@@ -47,4 +47,14 @@ qemu-img: Preallocation can only be used for growing images --- growth_mode=off --- Image resized. + +=== Testing image growth on 2G empty image === + +--- growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 +Image resized. + +--- growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 +Image resized. *** done
Signature · 1 line
2.14.3
Re: [Qemu-devel] [PATCH 2/2] iotests: Test preallocated truncate of 2G image
Quoted 50 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/106 | 24 ++++++++++++++++++++++++ tests/qemu-iotests/106.out | 10 ++++++++++ 2 files changed, 34 insertions(+)tests/qemu-iotests/106 +24
-0@@ -86,6 +86,30 @@ for growth_mode in falloc full off; do $QEMU_IMG resize -f "$IMGFMT" --shrink --preallocation=$growth_mode "$TEST_IMG" -${GROWTH_SIZE}K done +echo +echo '=== Testing image growth on 2G empty image ===' + +for growth_mode in falloc full; do + echo + echo "--- growth_mode=$growth_mode ---" + + # Maybe we want to do an lseek() to the end of the file before the + # preallocation; if the file has a length of 2 GB, that would + # return an integer that overflows to negative when put into a + # plain int. We should use the correct type for the result, and + # this tests we do. + + _make_test_img 2G + $QEMU_IMG resize -f "$IMGFMT" --preallocation=$growth_mode "$TEST_IMG" +${GROWTH_SIZE}K + + actual_size=$($QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep 'disk size') + actual_size=$(echo "$actual_size" | sed -e 's/^[^0-9]*\([0-9]\+\).*$/\1/') + + if [ $actual_size -lt $GROWTH_SIZE ]; then + echo "ERROR: Image should have at least ${GROWTH_SIZE}K, but has ${actual_size}K" + fi +done + # success, all done echo '*** done' rm -f $seq.fulltests/qemu-iotests/106.out +1
-0@@ -47,4 +47,14 @@ qemu-img: Preallocation can only be used for growing images --- growth_mode=off --- Image resized. +
tests/qemu-iotests/106.out qemu-img: Preallocation can only be used for growing images · line 60
Quoted 10 lines
tests/qemu-iotests/106.out +9
-0+=== Testing image growth on 2G empty image === + +--- growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 +Image resized. + +--- growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 +Image resized. *** done
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 8 lines · Max Reitz
Fully preallocated truncation has a 50 % chance of working on images files over file-posix. It works if $SIZE % 4G < 2G, and it fails otherwise. To make things even more interesting, often you would not even notice because qemu reported success even though it did nothing (because after the successful
lseek(), errno was still 0, so when the file-posix driver tried to return a negative error code, it actually reported success).
Quoted 7 lines
This issue is fixed by patch 1 in this series. Thanks to Daniel for reporting!
Max Reitz (2):
block/file-posix: Fix fully preallocated truncate
iotests: Test preallocated truncate of 2G image
Good catch. Series:
gluster.c has similar code, but it assigns glfs_lseek() to an int64_t, so it does not have the bug.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Max Reitz
AppliedQuoted 12 lines · Max Reitz
Fully preallocated truncation has a 50 % chance of working on images files over file-posix. It works if $SIZE % 4G < 2G, and it fails otherwise. To make things even more interesting, often you would not even notice because qemu reported success even though it did nothing (because after the successful
lseek(), errno was still 0, so when the file-posix driver tried to return a negative error code, it actually reported success).This issue is fixed by patch 1 in this series. Thanks to Daniel for reporting!
Quoted 8 lines
Max Reitz (2):
block/file-posix: Fix fully preallocated truncate
iotests: Test preallocated truncate of 2G imageblock/file-posix.c | 5 +++-- tests/qemu-iotests/106 | 24 ++++++++++++++++++++++++ tests/qemu-iotests/106.out | 10 ++++++++++ 3 files changed, 37 insertions(+), 2 deletions(-)
Applied to my block tree... (*cough* *cough* -- I'm the worst when it comes to keeping track of my own patches.)
Max