block: posix: Always allocate the first block
PATCH 1/1 block: posix: Always allocate the first block
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
When using preallocation=off, we always allocate at least one filesystem block:
$ ./qemu-img create -f raw test.raw 1g
Formatting 'test.raw', fmt=raw size=1073741824 $ ls -lhs test.raw
4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawI did quick performance tests for these flows:
- Provisioning a VM with a new raw image.
- Copying disks with qemu-img convert to new raw target image
I installed Fedora 29 server on raw sparse image, measuring the time from clicking "Begin installation" until the "Reboot" button appears:
Before(s) After(s) Diff(%)
-------------------------------
356 389 +8.4
I ran this only once, so we cannot tell much from these results.
The second test was cloning the installation image with qemu-img convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
done
Here is a table comparing the total time spent:
Type Before(s) After(s) Diff(%)
---------------------------------------
real 530.028 469.123 -11.4 user 17.204 10.768 -37.4 sys 17.881 7.011 -60.7
Here we see very clear improvement in CPU usage.
--- block/file-posix.c | 25 +++++++++++++++++++++++++ tests/qemu-iotests/150.out | 1 + tests/qemu-iotests/160 | 4 ++++ tests/qemu-iotests/175 | 19 +++++++++++++------ tests/qemu-iotests/175.out | 8 ++++---- tests/qemu-iotests/221.out | 12 ++++++++---- tests/qemu-iotests/253.out | 12 ++++++++---- 7 files changed, 63 insertions(+), 18 deletions(-)
block/file-posix.c +25 -0
@@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void *opaque)return ret; } +/* +* Help alignment detection by allocating the first block. +* +* When reading with direct I/O from unallocated area on Gluster backed by XFS, +* reading succeeds regardless of request length. In this case we fallback to +* safe aligment which is not optimal. Allocating the first block avoids this +* fallback. +* +* Returns: 0 on success, -errno on failure. +*/ +static int allocate_first_block (int fd) +{ +ssize_t n; + +do { + n =pwrite (fd,"\0" ,1 ,0 ); + }while (n ==-1 && errno ==EINTR ); + +return (n ==-1 ) ? -errno :0 ; +} +static int handle_aiocb_truncate (void *opaque) {RawPosixAIOData *aiocb = opaque;
@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void *opaque)/* posix_fallocate() doesn't set errno. */ error_setg_errno (errp, -result,"Could not preallocate new data" ); + }else if (current_length ==0 ) { +allocate_first_block (fd); } }else { result =0 ;
@@ -1855,6 +1878,8 @@ static int handle_aiocb_truncate(void *opaque)if (ftruncate (fd, offset) !=0 ) { result = -errno;error_setg_errno (errp, -result,"Could not resize file" ); + }else if (current_length ==0 && offset > current_length) { +allocate_first_block (fd); }return result;default :
tests/qemu-iotests/150.out +1 -0
@@ -3,6 +3,7 @@ QA output created by 150 === Mapping sparse conversion === Offset Length File +0 0x1000 TEST_DIR/t.IMGFMT === Mapping non-sparse conversion ===
tests/qemu-iotests/160 +4 -0
@@ -57,6 +57,10 @@ for skip in $TEST_SKIP_BLOCKS; do $QEMU_IMG dd if="$TEST_IMG" of="$TEST_IMG.out" skip="$skip" -O "$IMGFMT" \ 2> /dev/null TEST_IMG="$TEST_IMG.out" _check_test_img + + # We always write the first byte of an image. + printf "\0" > "$TEST_IMG.out.dd" + dd if="$TEST_IMG" of="$TEST_IMG.out.dd" skip="$skip" status=none echo
tests/qemu-iotests/175 +13 -6
@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # the file size. This function hides the resulting difference in the # stat -c '%b' output. # Parameter 1: Number of blocks an empty file occupies -# Parameter 2: Image size in bytes +# Parameter 2: Minimal number of blocks in an image +# Parameter 3: Image size in bytes _filter_blocks() { extra_blocks=$1 - img_size=$2 + min_blocks=$2 + img_size=$3 - sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \ - -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/" + sed -e "s/blocks=$((extra_blocks + min_blocks))\\(\$\\|[^0-9]\\)/min allocation/" \ + -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/" } # get standard environment, filters and checks
@@ -60,16 +62,21 @@ size=$((1 * 1024 * 1024)) touch "$TEST_DIR/empty" extra_blocks=$(stat -c '%b' "$TEST_DIR/empty") +# We always write the first byte; check how many blocks this filesystem +# allocates to match empty image alloation. +printf "\0" > "$TEST_DIR/empty" +min_blocks=$(stat -c '%b' "$TEST_DIR/empty") + echo echo "== creating image with default preallocation ==" _make_test_img $size | _filter_imgfmt -stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size +stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size for mode in off full falloc; do echo echo "== creating image with preallocation $mode ==" IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt - stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size + stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size done # success, all done
tests/qemu-iotests/175.out +4 -4
@@ -2,17 +2,17 @@ QA output created by 175 == creating image with default preallocation == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 -size=1048576, nothing allocated +size=1048576, min allocation == creating image with preallocation off == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off -size=1048576, nothing allocated +size=1048576, min allocation == creating image with preallocation full == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full -size=1048576, everything allocated +size=1048576, max allocation == creating image with preallocation falloc == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc -size=1048576, everything allocated +size=1048576, max allocation *** done
tests/qemu-iotests/221.out +8 -4
@@ -3,14 +3,18 @@ QA output created by 221 === Check mapping of unaligned raw image === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] wrote 1/1 bytes at offset 65536 1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] *** done
tests/qemu-iotests/253.out +8 -4
@@ -3,12 +3,16 @@ QA output created by 253 === Check mapping of unaligned raw image === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] wrote 65535/65535 bytes at offset 983040 63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
Signature · 1 line
2.20.1
Quoted 9 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
Where does this detection/fallback happen? (Can it be improved?)
Quoted 13 lines
When using preallocation=off, we always allocate at least one filesystem block:
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawI did quick performance tests for these flows:
- Provisioning a VM with a new raw image.
- Copying disks with qemu-img convert to new raw target image
Quoted 9 lines
I installed Fedora 29 server on raw sparse image, measuring the time from clicking "Begin installation" until the "Reboot" button appears:
Before(s) After(s) Diff(%)
-------------------------------356 389 +8.4I ran this only once, so we cannot tell much from these results.
That seems like a pretty big difference for just having pre-allocated a single block. What was the actual command line / block graph for that test?
Was this over a network that could explain the variance?
Quoted 11 lines
The second test was cloning the installation image with qemu-img convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
doneHere is a table comparing the total time spent:
Quoted 8 lines
Type Before(s) After(s) Diff(%)
---------------------------------------real 530.028 469.123 -11.4 user 17.204 10.768 -37.4 sys 17.881 7.011 -60.7Here we see very clear improvement in CPU usage.
Hard to argue much with that. I feel a little strange trying to force the allocation of the first block, but I suppose in practice "almost no preallocation" is indistinguishable from "exactly no preallocation" if you squint.
Quoted 216 lines
Signed-off-by Nir Soffer <[email protected]>--- block/file-posix.c | 25 +++++++++++++++++++++++++ tests/qemu-iotests/150.out | 1 + tests/qemu-iotests/160 | 4 ++++ tests/qemu-iotests/175 | 19 +++++++++++++------ tests/qemu-iotests/175.out | 8 ++++---- tests/qemu-iotests/221.out | 12 ++++++++---- tests/qemu-iotests/253.out | 12 ++++++++---- 7 files changed, 63 insertions(+), 18 deletions(-)block/file-posix.c +25
-0@@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void *opaque)return ret; } +/* +* Help alignment detection by allocating the first block. +* +* When reading with direct I/O from unallocated area on Gluster backed by XFS, +* reading succeeds regardless of request length. In this case we fallback to +* safe aligment which is not optimal. Allocating the first block avoids this +* fallback. +* +* Returns: 0 on success, -errno on failure. +*/ +static int allocate_first_block (int fd) +{ +ssize_t n; + +do { + n =pwrite (fd,"\0" ,1 ,0 ); + }while (n ==-1 && errno ==EINTR ); + +return (n ==-1 ) ? -errno :0 ; +} +static int handle_aiocb_truncate (void *opaque) {RawPosixAIOData *aiocb = opaque;@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void *opaque)/* posix_fallocate() doesn't set errno. */ error_setg_errno (errp, -result,"Could not preallocate new data" ); + }else if (current_length ==0 ) { +allocate_first_block (fd); } }else { result =0 ;@@ -1855,6 +1878,8 @@ static int handle_aiocb_truncate(void *opaque)if (ftruncate (fd, offset) !=0 ) { result = -errno;error_setg_errno (errp, -result,"Could not resize file" ); + }else if (current_length ==0 && offset > current_length) { +allocate_first_block (fd); }return result;default :tests/qemu-iotests/150.out +1
-0@@ -3,6 +3,7 @@ QA output created by 150 === Mapping sparse conversion === Offset Length File +0 0x1000 TEST_DIR/t.IMGFMT === Mapping non-sparse conversion ===tests/qemu-iotests/160 +4
-0@@ -57,6 +57,10 @@ for skip in $TEST_SKIP_BLOCKS; do $QEMU_IMG dd if="$TEST_IMG" of="$TEST_IMG.out" skip="$skip" -O "$IMGFMT" \ 2> /dev/null TEST_IMG="$TEST_IMG.out" _check_test_img + + # We always write the first byte of an image. + printf "\0" > "$TEST_IMG.out.dd" + dd if="$TEST_IMG" of="$TEST_IMG.out.dd" skip="$skip" status=none echotests/qemu-iotests/175 +13
-6@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # the file size. This function hides the resulting difference in the # stat -c '%b' output. # Parameter 1: Number of blocks an empty file occupies -# Parameter 2: Image size in bytes +# Parameter 2: Minimal number of blocks in an image +# Parameter 3: Image size in bytes _filter_blocks() { extra_blocks=$1 - img_size=$2 + min_blocks=$2 + img_size=$3 - sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \ - -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/" + sed -e "s/blocks=$((extra_blocks + min_blocks))\\(\$\\|[^0-9]\\)/min allocation/" \ + -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/" } # get standard environment, filters and checks@@ -60,16 +62,21 @@ size=$((1 * 1024 * 1024)) touch "$TEST_DIR/empty" extra_blocks=$(stat -c '%b' "$TEST_DIR/empty") +# We always write the first byte; check how many blocks this filesystem +# allocates to match empty image alloation. +printf "\0" > "$TEST_DIR/empty" +min_blocks=$(stat -c '%b' "$TEST_DIR/empty") + echo echo "== creating image with default preallocation ==" _make_test_img $size | _filter_imgfmt -stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size +stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size for mode in off full falloc; do echo echo "== creating image with preallocation $mode ==" IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt - stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size + stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size done # success, all donetests/qemu-iotests/175.out +4
-4@@ -2,17 +2,17 @@ QA output created by 175 == creating image with default preallocation == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 -size=1048576, nothing allocated +size=1048576, min allocation == creating image with preallocation off == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off -size=1048576, nothing allocated +size=1048576, min allocation == creating image with preallocation full == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full -size=1048576, everything allocated +size=1048576, max allocation == creating image with preallocation falloc == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc -size=1048576, everything allocated +size=1048576, max allocation *** donetests/qemu-iotests/221.out +8
-4@@ -3,14 +3,18 @@ QA output created by 221 === Check mapping of unaligned raw image === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] wrote 1/1 bytes at offset 65536 1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] *** donetests/qemu-iotests/253.out +8
-4@@ -3,12 +3,16 @@ QA output created by 253 === Check mapping of unaligned raw image === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] wrote 65535/65535 bytes at offset 983040 63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
Quoted 6 lines · John Snow
Quoted 5 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
Quoted 7 lines
Quoted 4 lines
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
Where does this detection/fallback happen? (Can it be improved?)
In raw_probe_alignment().
This patch explain the issues:
https://lists.nongnu.org/archive/html/qemu-block/2019-08/msg00568.html
Here Kevin and me discussed ways to improve it:
https://lists.nongnu.org/archive/html/qemu-block/2019-08/msg00426.html
Quoted 20 lines
When using preallocation=off, we always allocate at least one filesystem
Quoted 19 lines
block:
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawI did quick performance tests for these flows:
- Provisioning a VM with a new raw image.
- Copying disks with qemu-img convert to new raw target imageI installed Fedora 29 server on raw sparse image, measuring the time from clicking "Begin installation" until the "Reboot" button appears:
Before(s) After(s) Diff(%)
-------------------------------356 389 +8.4
Quoted 6 lines
Quoted 2 lines
I ran this only once, so we cannot tell much from these results.
That seems like a pretty big difference for just having pre-allocated a single block. What was the actual command line / block graph for that test?
Having the first block allocated changes the alignment.
Before this patch, we detect request_alignment=1, so we fallback to 4096. Then we detect buf_align=1, so we fallback to value of request alignment.
The guest see a disk with:logical_block_size = 512physical_block_size = 512
But qemu uses:request_alignment = 4096buf_align = 4096
storage uses:logical_block_size = 512physical_block_size = 512
If the guest does direct I/O using 512 bytes aligment, qemu has to copy the buffer to align them to 4096 bytes.
After this patch, qemu detects the alignment correctly, so we have:
guestlogical_block_size = 512physical_block_size = 512
qemurequest_alignment = 512buf_align = 512
storage:logical_block_size = 512physical_block_size = 512
We expect this to be more efficient because qemu does not have to emulate anything.
Was this over a network that could explain the variance?
Quoted 1 line
Maybe, this is complete install of Fedora 29 server, I'm not sure if the installation
access the network.
Quoted 18 lines
The second test was cloning the installation image with qemu-img
Quoted 6 lines
convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T none src.rawdst.raw
Quoted 10 lines
done
Here is a table comparing the total time spent:
Type Before(s) After(s) Diff(%)
---------------------------------------real 530.028 469.123 -11.4 user 17.204 10.768 -37.4 sys 17.881 7.011 -60.7
Quoted 8 lines
Quoted 2 lines
Here we see very clear improvement in CPU usage.
Hard to argue much with that. I feel a little strange trying to force the allocation of the first block, but I suppose in practice "almost no preallocation" is indistinguishable from "exactly no preallocation" if you squint.
Right.
The real issue is that filesystems and block devices do not expose the alignment
requirement for direct I/O, so we need to use these hacks and assumptions.
With local XFS we use xfsctl(XFS_IOC_DIOINFO) to get request_alignment, but this does
not help for XFS filesystem used by Gluster on the server side.
I hope that Niels is working on adding similar ioctl for Glsuter, os it can expose the properties
of the remote filesystem.
Nir
Quoted 17 lines · Nir Soffer
On Sat, Aug 17, 2019 at 12:57 AM John Snow <[email protected] <mailto:[email protected]>> wrote:
On 8/16/19 5:21 PM, Nir Soffer wrote:
> When creating an image with preallocation "off" or "falloc", the first > block of the image is typically not allocated. When using Gluster > storage backed by XFS filesystem, reading this block using direct I/O > succeeds regardless of request length, fooling alignment detection. >
> In this case we fallback to a safe value (4096) instead of the optimal > value (512), which may lead to unneeded data copying when aligning > requests. Allocating the first block avoids the fallback. >Where does this detection/fallback happen? (Can it be improved?)
Quoted 8 lines
In raw_probe_alignment().
This patch explain the issues:
https://lists.nongnu.org/archive/html/qemu-block/2019-08/msg00568.htmlHere Kevin and me discussed ways to improve it:
https://lists.nongnu.org/archive/html/qemu-block/2019-08/msg00426.html
Thanks for the reading!
That does help explain this patch better.
Quoted 110 lines
Quoted 1 line
When using preallocation=off, we always allocate at least one
filesystem
> block:
>
> $ ./qemu-img create -f raw test.raw 1g
> Formatting 'test.raw', fmt=raw size=1073741824
>
> $ ls -lhs test.raw
> 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw >
> I did quick performance tests for these flows:
> - Provisioning a VM with a new raw image.
> - Copying disks with qemu-img convert to new raw target image >
> I installed Fedora 29 server on raw sparse image, measuring the time > from clicking "Begin installation" until the "Reboot" button appears:
>
> Before(s) After(s) Diff(%)
> -------------------------------
> 356 389 +8.4
>
> I ran this only once, so we cannot tell much from these results. >That seems like a pretty big difference for just having pre-allocated a single block. What was the actual command line / block graph for that test?
Having the first block allocated changes the alignment.
Before this patch, we detect request_alignment=1, so we fallback to 4096. Then we detect buf_align=1, so we fallback to value of request alignment.
The guest see a disk with:
logical_block_size= 512physical_block_size= 512But qemu uses:
request_alignment= 4096buf_align= 4096storage uses:
logical_block_size= 512physical_block_size= 512If the guest does direct I/O using 512 bytes aligment, qemu has to copy the buffer to align them to 4096 bytes.
After this patch, qemu detects the alignment correctly, so we have:
guest
logical_block_size= 512physical_block_size= 512qemu
request_alignment= 512buf_align= 512storage:
logical_block_size= 512physical_block_size= 512We expect this to be more efficient because qemu does not have to emulate anything.
Was this over a network that could explain the variance?
Maybe, this is complete install of Fedora 29 server, I'm not sure if the installation
access the network.Quoted 7 lines
The second test was cloning the installation image with qemu-img convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T nonesrc.raw dst.raw > done > > Here is a table comparing the total time spent: > > Type Before(s) After(s) Diff(%) > --------------------------------------- > real 530.028 469.123 -11.4 > user 17.204 10.768 -37.4 > sys 17.881 7.011 -60.7 > > Here we see very clear improvement in CPU usage. > Hard to argue much with that. I feel a little strange trying to force the allocation of the first block, but I suppose in practice "almost no preallocation" is indistinguishable from "exactly no preallocation" if you squint.Right.
The real issue is that filesystems and block devices do not expose the alignment
requirement for direct I/O, so we need to use these hacks and assumptions.With local XFS we use
xfsctl(XFS_IOC_DIOINFO)to getrequest_alignment, but this does
not help for XFS filesystem used by Gluster on the server side.
Quoted 5 lines
I hope that Niels is working on adding similar ioctl for Glsuter, os it can expose the properties
of the remote filesystem.Nir
That sounds quite a bit less hacky, but I agree we still have to do what we can in the meantime.
(It looks like you've been hashing this out with Kevin for a while, so I'm going to sheepishly defer to his judgment on this patch. While I think it's probably a fine trade-off, I can't really say off-hand if there's a better, more targeted way to accomplish it.)
--js
Max, did you have time to look at this?
Quoted 312 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
When using preallocation=off, we always allocate at least one filesystem block:
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawI did quick performance tests for these flows:
- Provisioning a VM with a new raw image.
- Copying disks with qemu-img convert to new raw target imageI installed Fedora 29 server on raw sparse image, measuring the time from clicking "Begin installation" until the "Reboot" button appears:
Before(s) After(s) Diff(%)
-------------------------------356 389 +8.4I ran this only once, so we cannot tell much from these results.
The second test was cloning the installation image with qemu-img convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T none src.raw
dst.raw
doneHere is a table comparing the total time spent:
Type Before(s) After(s) Diff(%)
---------------------------------------real 530.028 469.123 -11.4 user 17.204 10.768 -37.4 sys 17.881 7.011 -60.7Here we see very clear improvement in CPU usage.
Signed-off-by Nir Soffer <[email protected]>--- block/file-posix.c | 25 +++++++++++++++++++++++++ tests/qemu-iotests/150.out | 1 + tests/qemu-iotests/160 | 4 ++++ tests/qemu-iotests/175 | 19 +++++++++++++------ tests/qemu-iotests/175.out | 8 ++++---- tests/qemu-iotests/221.out | 12 ++++++++---- tests/qemu-iotests/253.out | 12 ++++++++---- 7 files changed, 63 insertions(+), 18 deletions(-)block/file-posix.c +5
-0@@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void *opaque)return ret; } +/* +* Help alignment detection by allocating the first block. +* +* When reading with direct I/O from unallocated area on Gluster backed by XFS, +* reading succeeds regardless of request length. In this case we fallback to
+ * safe aligment which is not optimal. Allocating the first block avoids this
+ * fallback.
+ *
+ * Returns: 0 on success, -errno on failure.
+ */block/file-posix.c +15
-0+static int allocate_first_block (int fd) +{ +ssize_t n; + +do { + n =pwrite (fd,"\0" ,1 ,0 ); + }while (n ==-1 && errno ==EINTR ); + +return (n ==-1 ) ? -errno :0 ; +} +static int handle_aiocb_truncate (void *opaque) {RawPosixAIOData *aiocb = opaque;@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void *opaque)/* posix_fallocate() doesn't set errno. */ error_setg_errno (errp, -result,"Could not preallocate new data" ); + }else if (current_length ==0 ) { +allocate_first_block (fd); } }else { result =0 ;@@ -1855,6 +1878,8 @@ static int handle_aiocb_truncate(void *opaque)if (ftruncate (fd, offset) !=0 ) { result = -errno;error_setg_errno (errp, -result,"Could not resize file" ); + }else if (current_length ==0 && offset > current_length) { +allocate_first_block (fd); }return result;default :tests/qemu-iotests/150.out +1
-0@@ -3,6 +3,7 @@ QA output created by 150 === Mapping sparse conversion === Offset Length File +0 0x1000 TEST_DIR/t.IMGFMT === Mapping non-sparse conversion ===tests/qemu-iotests/160 +4
-0@@ -57,6 +57,10 @@ for skip in $TEST_SKIP_BLOCKS; do $QEMU_IMG dd if="$TEST_IMG" of="$TEST_IMG.out" skip="$skip" -O "$IMGFMT" \ 2> /dev/null TEST_IMG="$TEST_IMG.out" _check_test_img + + # We always write the first byte of an image. + printf "\0" > "$TEST_IMG.out.dd" + dd if="$TEST_IMG" of="$TEST_IMG.out.dd" skip="$skip" status=noneecho
tests/qemu-iotests/175 +4
-4@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # the file size. This function hides the resulting difference in the # stat -c '%b' output. # Parameter 1: Number of blocks an empty file occupies -# Parameter 2: Image size in bytes +# Parameter 2: Minimal number of blocks in an image +# Parameter 3: Image size in bytes _filter_blocks() { extra_blocks=$1 - img_size=$2 + min_blocks=$2 + img_size=$3 - sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \ - -e "s/blocks=$((extra_blocks + img_size /512))\\(\$\\|[^0-9]\\)/everything allocated/"
+ sed -e "s/blocks=$((extra_blocks + min_blocks))\\(\$\\|[^0-9]\\)/min
allocation/" \
+ -e "s/blocks=$((extra_blocks +img_size/
512))\\(\$\\|[^0-9]\\)/max allocation/"
}# get standard environment, filters and checkstests/qemu-iotests/175 +5
-1@@ -60,16 +62,21 @@ size=$((1 * 1024 * 1024)) touch "$TEST_DIR/empty" extra_blocks=$(stat -c '%b' "$TEST_DIR/empty") +# We always write the first byte; check how many blocks this filesystem +# allocates to match empty image alloation. +printf "\0" > "$TEST_DIR/empty" +min_blocks=$(stat -c '%b' "$TEST_DIR/empty") + echo echo "== creating image with default preallocation ==" _make_test_img $size | _filter_imgfmt -stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks$size
+stat -c "size=%s, blocks=%b" $TEST_IMG |_filter_blocks$extra_blocks $min_blocks $sizefor mode in off full falloc; do
echo
echo "== creating image with preallocation $mode ==" IMGOPTS=preallocation=$mode_make_test_img$size |_filter_imgfmt
- stat -c "size=%s, blocks=%b" $TEST_IMG |_filter_blocks$extra_blocks
$size
+ stat -c "size=%s, blocks=%b" $TEST_IMG |_filter_blocks$extra_blocks
$min_blocks $size
done# success, all done
tests/qemu-iotests/175.out +3
-3@@ -2,17 +2,17 @@ QA output created by 175 == creating image with default preallocation == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 -size=1048576, nothing allocated +size=1048576, min allocation == creating image with preallocation off == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off -size=1048576, nothing allocated +size=1048576, min allocation == creating image with preallocation full == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full -size=1048576, everything allocated +size=1048576, max allocation == creating image with preallocation falloc == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576preallocation=falloc
tests/qemu-iotests/175.out +1
-1-size=1048576, everything allocated +size=1048576, max allocation *** donetests/qemu-iotests/221.out +0
-1@@ -3,14 +3,18 @@ QA output created by 221 === Check mapping of unaligned raw image === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false,"offset": OFFSET}]
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data":
false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data":
false, "offset": OFFSET}]
wrote 1/1 bytes at offset 65536
1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data":
false, "offset": OFFSET},
{ "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true,
"offset": OFFSET},
{ "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false,
"offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data":
false, "offset": OFFSET},
{ "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true,
"offset": OFFSET},
{ "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false,
"offset": OFFSET}]
*** donetests/qemu-iotests/253.out +0
-1@@ -3,12 +3,16 @@ QA output created by 253 === Check mapping of unaligned raw image === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data":false, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data":
false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data":
false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data":
false, "offset": OFFSET}]
wrote 65535/65535 bytes at offset 983040
63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data":
false, "offset": OFFSET},
{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data":
true, "offset": OFFSET}]
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data":
false, "offset": OFFSET},
{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data":
true, "offset": OFFSET}]
*** done
--
2.20.1
Quoted 22 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
When using preallocation=off, we always allocate at least one filesystem block:
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawI did quick performance tests for these flows:
- Provisioning a VM with a new raw image.
- Copying disks with qemu-img convert to new raw target image
Quoted 8 lines
I installed Fedora 29 server on raw sparse image, measuring the time from clicking "Begin installation" until the "Reboot" button appears:
Before(s) After(s) Diff(%)
-------------------------------356 389 +8.4I ran this only once, so we cannot tell much from these results.
So you’d expect it to be fast but it was slower? Well, you only ran it once and it isn’t really a precise benchmark...
Quoted 44 lines
The second test was cloning the installation image with qemu-img convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
doneHere is a table comparing the total time spent:
Type Before(s) After(s) Diff(%)
---------------------------------------real 530.028 469.123 -11.4 user 17.204 10.768 -37.4 sys 17.881 7.011 -60.7Here we see very clear improvement in CPU usage.
Signed-off-by Nir Soffer <[email protected]>--- block/file-posix.c | 25 +++++++++++++++++++++++++ tests/qemu-iotests/150.out | 1 + tests/qemu-iotests/160 | 4 ++++ tests/qemu-iotests/175 | 19 +++++++++++++------ tests/qemu-iotests/175.out | 8 ++++---- tests/qemu-iotests/221.out | 12 ++++++++---- tests/qemu-iotests/253.out | 12 ++++++++---- 7 files changed, 63 insertions(+), 18 deletions(-)block/file-posix.c +6
-0@@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void *opaque)return ret; } +/* +* Help alignment detection by allocating the first block. +* +* When reading with direct I/O from unallocated area on Gluster backed by XFS, +* reading succeeds regardless of request length. In this case we fallback to +* safe aligment which is not optimal. Allocating the first block avoids this
block/file-posix.c static int handle_aiocb_discard(void *opaque) · line 1773
Quoted 10 lines
block/file-posix.c +10
-0+* fallback. +* +* Returns: 0 on success, -errno on failure. +*/ +static int allocate_first_block (int fd) +{ +ssize_t n; + + do { + n = pwrite(fd, "\0", 1, 0);
This breaks when fd has been opened with O_DIRECT.
(Which happens when you open some file with cache.direct=on, and then use e.g. QMP’s block_resize.)
It isn’t that bad because eventually you simply ignore the error. But it still makes me wonder whether we shouldn’t write like the biggest power of two that does not exceed the new file length or MAX_BLOCKSIZE.
Quoted 9 lines
block/file-posix.c +5
-0+ }while (n ==-1 && errno ==EINTR ); + +return (n ==-1 ) ? -errno :0 ; +} +static int handle_aiocb_truncate (void *opaque) {RawPosixAIOData *aiocb = opaque;
block/file-posix.c static int handle_aiocb_truncate(void *opaque) · line 1819
Quoted 5 lines
block/file-posix.c +2
-0@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void *opaque)/* posix_fallocate() doesn't set errno. */ error_setg_errno (errp, -result,"Could not preallocate new data" ); + }else if (current_length ==0 ) { +allocate_first_block (fd);
Should posix_fallocate() not take care of precisely this?
Quoted 3 lines
block/file-posix.c +0
-0} }else { result =0 ;
[...]
tests/qemu-iotests/160 for skip in $TEST_SKIP_BLOCKS; do · line 64
Quoted 13 lines
tests/qemu-iotests/160 +4
-0@@ -57,6 +57,10 @@ for skip in $TEST_SKIP_BLOCKS; do $QEMU_IMG dd if="$TEST_IMG" of="$TEST_IMG.out" skip="$skip" -O "$IMGFMT" \ 2> /dev/null TEST_IMG="$TEST_IMG.out" _check_test_img + + # We always write the first byte of an image. + printf "\0" > "$TEST_IMG.out.dd" + dd if="$TEST_IMG" of="$TEST_IMG.out.dd" skip="$skip" status=none
Won’t this dd completely overwrite $TEST_IMG.out.dd (especially given the lack of conv=notrunc)?
Quoted 13 lines
tests/qemu-iotests/160 +0
-0echotests/qemu-iotests/175 +2
-1@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # the file size. This function hides the resulting difference in the # stat -c '%b' output. # Parameter 1: Number of blocks an empty file occupies -# Parameter 2: Image size in bytes +# Parameter 2: Minimal number of blocks in an image +# Parameter 3: Image size in bytes
tests/qemu-iotests/175 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 48
Quoted 10 lines
tests/qemu-iotests/175 +3
-3_filter_blocks() { extra_blocks=$1 - img_size=$2 + min_blocks=$2 + img_size=$3 - sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \ - -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/" + sed -e "s/blocks=$((extra_blocks + min_blocks))\\(\$\\|[^0-9]\\)/min allocation/" \
I don’t think adding extra_blocks to min_blocks makes sense. Just min_blocks alone should be what we want here.
Max
Quoted 83 lines · Max Reitz
Quoted 22 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
When using preallocation=off, we always allocate at least one filesystem block:
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawI did quick performance tests for these flows:
- Provisioning a VM with a new raw image.
- Copying disks with qemu-img convert to new raw target imageQuoted 8 lines
I installed Fedora 29 server on raw sparse image, measuring the time from clicking "Begin installation" until the "Reboot" button appears:
Before(s) After(s) Diff(%)
-------------------------------356 389 +8.4I ran this only once, so we cannot tell much from these results.
So you’d expect it to be fast but it was slower? Well, you only ran it once and it isn’t really a precise benchmark...
Quoted 7 lines
The second test was cloning the installation image with qemu-img convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T none src.rawdst.raw
Quoted 28 lines
done
Here is a table comparing the total time spent:
Type Before(s) After(s) Diff(%)
---------------------------------------real 530.028 469.123 -11.4 user 17.204 10.768 -37.4 sys 17.881 7.011 -60.7Here we see very clear improvement in CPU usage.
Signed-off-by Nir Soffer <[email protected]>--- block/file-posix.c | 25 +++++++++++++++++++++++++ tests/qemu-iotests/150.out | 1 + tests/qemu-iotests/160 | 4 ++++ tests/qemu-iotests/175 | 19 +++++++++++++------ tests/qemu-iotests/175.out | 8 ++++---- tests/qemu-iotests/221.out | 12 ++++++++---- tests/qemu-iotests/253.out | 12 ++++++++---- 7 files changed, 63 insertions(+), 18 deletions(-)block/file-posix.c +0
-0block/file-posix.c static int handle_aiocb_discard(void *opaque) · line 1761
Quoted 7 lines
block/file-posix.c +4
-0@@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void *opaque)return ret; } +/* +* Help alignment detection by allocating the first block. +* +* When reading with direct I/O from unallocated area on Gluster backed by XFS,
block/file-posix.c static int handle_aiocb_discard(void *opaque) · line 1762
Quoted 1 line
block/file-posix.c +1
-0+* reading succeeds regardless of request length. In this case we fallback to
block/file-posix.c static int handle_aiocb_discard(void *opaque) · line 1763
Quoted 1 line
block/file-posix.c +1
-0+* safe aligment which is not optimal. Allocating the first block avoids this
Quoted 13 lines
block/file-posix.c static int handle_aiocb_discard(void *opaque) · line 1773
Quoted 10 lines
block/file-posix.c +10
-0+* fallback. +* +* Returns: 0 on success, -errno on failure. +*/ +static int allocate_first_block (int fd) +{ +ssize_t n; + + do { + n = pwrite(fd, "\0", 1, 0);This breaks when fd has been opened with
O_DIRECT.
It seems that we always open images without O_DIRECT when creating an image in qemu-img create, or when creating a target image in qemu-img convert.
Here is a trace of qemu-img create:
$ strace -f -tt -o /tmp/create.trace ./qemu-img create -f raw -o preallocation=falloc /tmp/gv1/src.raw 1g Formatting '/tmp/gv1/src.raw', fmt=raw size=1073741824 preallocation=falloc
1. open #1
17686 18:58:23.681921 openat(AT_FDCWD, "/tmp/gv1/src.raw", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 9
17686 18:58:23.683753 fstat(9, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0
17686 18:58:23.683829 close(9) = 0
2. open #2
17686 18:58:23.684146 openat(AT_FDCWD, "/tmp/gv1/src.raw", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 9
17686 18:58:23.684227 fstat(9, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0
17686 18:58:23.684256 close(9) = 0
3. open #3
17686 18:58:23.684339 openat(AT_FDCWD, "/tmp/gv1/src.raw", O_RDWR|O_CREAT|O_CLOEXEC, 0644) = 9
...
17688 18:58:23.690178 fstat(9, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0
17688 18:58:23.690217 ftruncate(9, 0 <unfinished ...>
...
17688 18:58:23.700266 <... ftruncate resumed>) = 0
...
17688 18:58:23.700595 fstat(9, <unfinished ...>
...
17688 18:58:23.700619 <... fstat resumed>{st_mode=S_IFREG|0600, st_size=0, ...}) = 0
...
17688 18:58:23.700651 fallocate(9, 0, 0, 1073741824 <unfinished ...>
...
17688 18:58:23.728141 <... fallocate resumed>) = 0
...
17688 18:58:23.728196 pwrite64(9, "\0", 1, 0) = 1
...
17686 18:58:23.738391 close(9) = 0
Here is convert trace:
$ strace -f -tt -o /tmp/convert.trace ./qemu-img convert -f raw -O raw -t none -T none /tmp/gv1/src.raw /tmp/gv1/dst.raw
1. open #1
18175 19:07:23.364417 openat(AT_FDCWD, "/tmp/gv1/dst.raw", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 10
18175 19:07:23.365282 fstat(10, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0
18175 19:07:23.365323 close(10) = 0
2. open #2
18175 19:07:23.365660 openat(AT_FDCWD, "/tmp/gv1/dst.raw", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 10
18175 19:07:23.365717 fstat(10, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0
18175 19:07:23.365742 close(10) = 0
3. open #3
18175 19:07:23.365839 openat(AT_FDCWD, "/tmp/gv1/dst.raw", O_RDWR|O_CREAT|O_CLOEXEC, 0644) = 10
...
18177 19:07:23.372112 fstat(10, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0
18177 19:07:23.372138 ftruncate(10, 0) = 0
...
18177 19:07:23.375760 fstat(10, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 18177 19:07:23.375788 ftruncate(10, 1073741824) = 0
18177 19:07:23.376383 pwrite64(10, "\0", 1, 0) = 1
...
18175 19:07:23.390989 close(10) = 0
4. open #4
18175 19:07:23.391429 openat(AT_FDCWD, "/tmp/gv1/dst.raw", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 10
18175 19:07:23.392433 fstat(10, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0
18175 19:07:23.392483 close(10) = 0
5. open #5
18175 19:07:23.392743 openat(AT_FDCWD, "/tmp/gv1/dst.raw", O_RDWR|O_DIRECT|O_CLOEXEC) = 10
...
18175 19:07:23.393731 ioctl(10, BLKSSZGET, 0x7ffe75ead334) = -1 ENOSYS (Function not implemented)
18175 19:07:23.393784 pread64(10, 0x558796451000, 1, 0) = -1 EINVAL (Invalid argument)
18175 19:07:23.395362 pread64(10, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512,
0) = 512
18175 19:07:23.395905 pread64(10, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 0) = 4096
...
(Which happens when you open some file with cache.direct=on, and then
Quoted 2 lines
use e.g. QMP’s
block_resize.)
What would be a command triggering this? I can add a test.
It isn’t that bad because eventually you simply ignore the error. But
Quoted 3 lines
it still makes me wonder whether we shouldn’t write like the biggest power of two that does not exceed the new file length or
MAX_BLOCKSIZE.
It makes sense if there is a way to cause qemu-img to use O_DIRECT when creating an image.
Quoted 17 lines
+ } while (n == -1 && errno == EINTR);
block/file-posix.c static int handle_aiocb_truncate(void *opaque) · line 1819
Quoted 13 lines
block/file-posix.c +6
-0+ +return (n ==-1 ) ? -errno :0 ; +} +static int handle_aiocb_truncate (void *opaque) {RawPosixAIOData *aiocb = opaque;@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void *opaque)/* posix_fallocate() doesn't set errno. */ error_setg_errno (errp, -result,"Could not preallocate new data" ); + }else if (current_length ==0 ) { +allocate_first_block (fd);Should
posix_fallocate()not take care of precisely this?
Only if the filesystem does not support fallocate() (e.g. NFS < 4.2).
In this case posix_fallocate() is doing:
for (offset += (len - 1) % increment; len > 0; offset += increment)
{
len -= increment;
if (offset < st.st_size)
{
unsigned char c;
ssize_t rsize = __pread (fd, &c, 1, offset);
if (rsize < 0)
return errno;
/* If there is a non-zero byte, the block must have been
allocated already. */
else if (rsize == 1 && c != 0)
continue;
}
if (__pwrite (fd, "", 1, offset) != 1)
return errno;
}https://code.woboq.org/userspace/glibc/sysdeps/posix/posix_fallocate.c.html#96
So opening a file with O_DIRECT will break preallocation=falloc on such filesystems,
and writing one byte in allocate_first_block() is safe.
Quoted 13 lines
Quoted 3 lines
block/file-posix.c +0
-0} }else { result =0 ;[...]
tests/qemu-iotests/160 for skip in $TEST_SKIP_BLOCKS; do · line 57
Quoted 6 lines
tests/qemu-iotests/160 +0
-0@@ -57,6 +57,10 @@ for skip in $TEST_SKIP_BLOCKS; do $QEMU_IMG dd if="$TEST_IMG" of="$TEST_IMG.out" skip="$skip" -O"$IMGFMT" \
Quoted 11 lines
tests/qemu-iotests/160 for skip in $TEST_SKIP_BLOCKS; do · line 64
Quoted 7 lines
tests/qemu-iotests/160 +4
-02> /dev/null TEST_IMG="$TEST_IMG.out" _check_test_img + + # We always write the first byte of an image. + printf "\0" > "$TEST_IMG.out.dd" + dd if="$TEST_IMG" of="$TEST_IMG.out.dd" skip="$skip" status=noneWon’t this dd completely overwrite $TEST_IMG.out.dd (especially given the lack of conv=notrunc)?
There is an issue only if dd open the file with O_TRUNC. I will test this again.
Quoted 22 lines
Quoted 11 lines
tests/qemu-iotests/160 +0
-0echotests/qemu-iotests/175 +0
-1@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # the file size. This function hides the resulting difference in the # stat -c '%b' output. # Parameter 1: Number of blocks an empty file occupies -# Parameter 2: Image size in bytestests/qemu-iotests/175 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 46
Quoted 10 lines
tests/qemu-iotests/175 +4
-2+# Parameter 2: Minimal number of blocks in an image +# Parameter 3: Image size in bytes _filter_blocks() { extra_blocks=$1 - img_size=$2 + min_blocks=$2 + img_size=$3 - sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/"\
Quoted 8 lines
tests/qemu-iotests/175 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 47
Quoted 1 line
tests/qemu-iotests/175 +0
-1- -e "s/blocks=$((extra_blocks + img_size /512))\\(\$\\|[^0-9]\\)/everything allocated/"
tests/qemu-iotests/175 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 48
Quoted 1 line
tests/qemu-iotests/175 +1
-0+ sed -e "s/blocks=$((extra_blocks +min_blocks))\\(\$\\|[^0-9]\\)/min allocation/" \
I don’t think adding
extra_blockstomin_blocksmakes sense. Justmin_blocksalone should be what we want here.
We had failing tests (in vdsm) showing that filesystem may return more blocs than
expected even for non-empty files, so this may be needed. I did not test it yet with a filesystem that show this issue, but I found your instructions how to create it.
Thanks for reviewing.
Nir
Quoted 102 lines · Nir Soffer
On Thu, Aug 22, 2019 at 5:28 PM Max Reitz <[email protected] <mailto:[email protected]>> wrote:
On 16.08.19 23:21, Nir Soffer wrote: > When creating an image with preallocation "off" or "falloc", the first > block of the image is typically not allocated. When using Gluster > storage backed by XFS filesystem, reading this block using direct I/O > succeeds regardless of request length, fooling alignment detection. > > In this case we fallback to a safe value (4096) instead of the optimal > value (512), which may lead to unneeded data copying when aligning > requests. Allocating the first block avoids the fallback. > > When using preallocation=off, we always allocate at least one filesystem > block: > > $ ./qemu-img create -f raw test.raw 1g > Formatting 'test.raw', fmt=raw size=1073741824 > > $ ls -lhs test.raw > 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw > > I did quick performance tests for these flows: > - Provisioning a VM with a new raw image. > - Copying disks with qemu-img convert to new raw target image > > I installed Fedora 29 server on raw sparse image, measuring the time > from clicking "Begin installation" until the "Reboot" button appears: > > Before(s) After(s) Diff(%) > ------------------------------- > 356 389 +8.4 > > I ran this only once, so we cannot tell much from these results. So you’d expect it to be fast but it was slower? Well, you only ran it once and it isn’t really a precise benchmark... > The second test was cloning the installation image with qemu-img > convert, doing 10 runs: > > for i in $(seq 10); do > rm -f dst.raw > sleep 10 > time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw > done > > Here is a table comparing the total time spent: > > Type Before(s) After(s) Diff(%) > --------------------------------------- > real 530.028 469.123 -11.4 > user 17.204 10.768 -37.4 > sys 17.881 7.011 -60.7 > > Here we see very clear improvement in CPU usage. > > Signed-off-by: Nir Soffer <[email protected] <mailto:[email protected]>> > --- > block/file-posix.c | 25 +++++++++++++++++++++++++ > tests/qemu-iotests/150.out | 1 + > tests/qemu-iotests/160 | 4 ++++ > tests/qemu-iotests/175 | 19 +++++++++++++------ > tests/qemu-iotests/175.out | 8 ++++---- > tests/qemu-iotests/221.out | 12 ++++++++---- > tests/qemu-iotests/253.out | 12 ++++++++---- > 7 files changed, 63 insertions(+), 18 deletions(-) > > diff --git a/block/file-posix.c b/block/file-posix.c > index b9c33c8f6c..3964dd2021 100644 > --- a/block/file-posix.c > +++ b/block/file-posix.c > @@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void *opaque) > return ret; > } > > +/* > + * Help alignment detection by allocating the first block. > + * > + * When reading with direct I/O from unallocated area on Gluster backed by XFS, > + * reading succeeds regardless of request length. In this case we fallback to > + * safe aligment which is not optimal. Allocating the first block avoids this > + * fallback. > + * > + * Returns: 0 on success, -errno on failure. > + */ > +static int allocate_first_block(int fd) > +{ > + ssize_t n; > + > + do { > + n = pwrite(fd, "\0", 1, 0); This breaks when fd has been opened with O_DIRECT.
Quoted 2 lines
It seems that we always open images without
O_DIRECTwhen creating an image in qemu-img create, or when creating a target image in qemu-img convert.
Yes. But you don’t call this function directly from image creation code but instead from the truncation function. (The former also calls the latter, but truncating is also an operation on its own.)
[...]
Quoted 5 lines
(Which happens when you open some file with cache.direct=on, and then use e.g. QMP’s
block_resize.)What would be a command triggering this? I can add a test.
block_resize, as I’ve said:
$ ./qemu-img create -f raw empty.img 0
$ x86_64-softmmu/qemu-system-x86_64 \
-qmp stdio \
-blockdev file,node-name=file,filename=empty.img,cache.direct=on \
<<EOF
{'execute':'qmp_capabilities'}
{'execute':'block_resize',
'arguments':{'node-name':'file',
'size':1048576}}
EOF
$ ./qemu-img map empty.img
Offset Length Mapped to File(You’d expect a data chunk here.)
I suppose you can get the same effect with blockdev-create and some format that explicitly resizes the file to some target length (LUKS does this, I think), but this is the most direct route.
Quoted 49 lines
It isn’t that bad because eventually you simply ignore the error. But it still makes me wonder whether we shouldn’t write like the biggest power of two that does not exceed the new file length or
MAX_BLOCKSIZE.It makes sense if there is a way to cause qemu-img to use
O_DIRECTwhen creating an image.Quoted 9 lines
+ } while (n == -1 && errno == EINTR);
block/file-posix.c +4
-0+ + return (n == -1) ? -errno : 0; +} +static int
handle_aiocb_truncate(void *opaque)
{
RawPosixAIOData *aiocb = opaque;block/file-posix.c static int handle_aiocb_truncate(void *opaque) · line 1819
Quoted 5 lines
block/file-posix.c +2
-0@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void *opaque) /* posix_fallocate() doesn't set errno. */ error_setg_errno(errp, -result, "Could not preallocate new data"); + } else if (current_length == 0) { + allocate_first_block(fd);Should
posix_fallocate()not take care of precisely this?Only if the filesystem does not support
fallocate()(e.g. NFS < 4.2).In this case
posix_fallocate()is doing:for (offset += (len - 1) % increment; len > 0; offset += increment) {
len -= increment;
if (offset < st.st_size)
{
unsigned char c;
ssize_trsize =__pread(fd, &c, 1, offset);
if (rsize < 0)
return errno;
/* If there is a non-zero byte, the block must have been allocated already. */
else if (rsize == 1 && c != 0)
continue;
}
if (__pwrite(fd, "", 1, offset) != 1)
return errno;
}
Quoted 4 lines
https://code.woboq.org/userspace/glibc/sysdeps/posix/posix_fallocate.c.html#96
So opening a file with
O_DIRECTwill break preallocation=falloc on such filesystems,
But won’t the function above just fail with EINVAL? allocate_first_block() is executed only in case of success.
Quoted 27 lines
and writing one byte in
allocate_first_block()is safe.block/file-posix.c static int handle_aiocb_truncate(void *opaque) · line 1822
Quoted 3 lines
block/file-posix.c +0
-0} } else { result = 0;[...] > diff --git a/tests/qemu-iotests/160 b/tests/qemu-iotests/160 > index df89d3864b..ad2d054a47 100755 > --- a/tests/qemu-iotests/160 > +++ b/tests/qemu-iotests/160 > @@ -57,6 +57,10 @@ for skip in $TEST_SKIP_BLOCKS; do > $QEMU_IMG dd if="$TEST_IMG" of="$TEST_IMG.out" skip="$skip" -O "$IMGFMT" \ > 2> /dev/null > TEST_IMG="$TEST_IMG.out" _check_test_img > + > + # We always write the first byte of an image. > + printf "\0" > "$TEST_IMG.out.dd" > + > dd if="$TEST_IMG" of="$TEST_IMG.out.dd" skip="$skip" status=none Won’t this dd completely overwrite $TEST_IMG.out.dd (especially given the lack of conv=notrunc)?
Quoted 1 line
There is an issue only if dd open the file with
O_TRUNC.
It isn’t an issue, I just don’t understand why the printf would be necessary at all.
dd should always truncate the output image unless conv=notrunc is specified. But even if it didn’t do that, in all of these test cases it should copy some data from $TEST_IMG to the output, and thus should always overwrite the first byte anyway.
Quoted 34 lines
I will test
this again.Quoted 15 lines
echotests/qemu-iotests/175 +2
-1@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # the file size. This function hides the resulting difference in the # stat -c '%b' output. # Parameter 1: Number of blocks an empty file occupies -# Parameter 2: Image size in bytes +# Parameter 2: Minimal number of blocks in an image +# Parameter 3: Image size in bytes _filter_blocks() {Quoted 6 lines
extra_blocks=$1
- img_size=$2
+ min_blocks=$2
+ img_size=$3
- sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothingallocated/" \ > - -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/" > + sed -e "s/blocks=$((extra_blocks + min_blocks))\\(\$\\|[^0-9]\\)/min allocation/" \ I don’t think adding extra_blocks to min_blocks makes sense. Just min_blocks alone should be what we want here.
Quoted 3 lines
We had failing tests (in vdsm) showing that filesystem may return more blocs than
expected even for non-empty files, so this may be needed.
But min_blocks is exactly the number of blocks of a file that has one allocated block. I don’t see how adding the number of blocks an empty file occupies makes sense.
Max
Quoted 124 lines · Max Reitz
Quoted 5 lines · Nir Soffer
On Thu, Aug 22, 2019 at 5:28 PM Max Reitz <[email protected] <mailto:[email protected]>> wrote:
On 16.08.19 23:21, Nir Soffer wrote:
> When creating an image with preallocation "off" or "falloc", thefirst
Quoted 2 lines
Quoted 2 lines
block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct
I/O
Quoted 3 lines
Quoted 3 lines
succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the
optimal
Quoted 10 lines
Quoted 4 lines
value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
When using preallocation=off, we always allocate at least one
filesystem
> block:
>
> $ ./qemu-img create -f raw test.raw 1g
> Formatting 'test.raw', fmt=raw size=1073741824
>Quoted 8 lines
> $ ls -lhs test.raw
> 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
>
> I did quick performance tests for these flows:
> - Provisioning a VM with a new raw image.
> - Copying disks with qemu-img convert to new raw target image >
> I installed Fedora 29 server on raw sparse image, measuring thetime
Quoted 1 line
Quoted 1 line
from clicking "Begin installation" until the "Reboot" button
appears:
Quoted 8 lines
Quoted 6 lines
Before(s) After(s) Diff(%)
-------------------------------356 389 +8.4I ran this only once, so we cannot tell much from these results.
So you’d expect it to be fast but it was slower? Well, you only ran
it
Quoted 31 lines
once and it isn’t really a precise benchmark... > The second test was cloning the installation image with qemu-img > convert, doing 10 runs: > > for i in $(seq 10); do > rm -f dst.raw > sleep 10 > time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw > done > > Here is a table comparing the total time spent: > > Type Before(s) After(s) Diff(%) > --------------------------------------- > real 530.028 469.123 -11.4 > user 17.204 10.768 -37.4 > sys 17.881 7.011 -60.7 > > Here we see very clear improvement in CPU usage. > > Signed-off-by: Nir Soffer <[email protected] <mailto:[email protected]>> > --- > block/file-posix.c | 25 +++++++++++++++++++++++++ > tests/qemu-iotests/150.out | 1 + > tests/qemu-iotests/160 | 4 ++++ > tests/qemu-iotests/175 | 19 +++++++++++++------ > tests/qemu-iotests/175.out | 8 ++++---- > tests/qemu-iotests/221.out | 12 ++++++++----Quoted 8 lines
> tests/qemu-iotests/253.out | 12 ++++++++---- > 7 files changed, 63 insertions(+), 18 deletions(-) > > diff --git a/block/file-posix.c b/block/file-posix.c > index b9c33c8f6c..3964dd2021 100644 > --- a/block/file-posix.c > +++ b/block/file-posix.c > @@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void*opaque)
Quoted 26 lines
Quoted 7 lines
return ret;}
+/*
+ * Help alignment detection by allocating the first block.
+ *
+ * When reading with direct I/O from unallocated area on Glusterbacked by XFS, > + * reading succeeds regardless of request length. In this case we fallback to > + * safe aligment which is not optimal. Allocating the first block avoids this > + * fallback. > + * > + * Returns: 0 on success, -errno on failure. > + */ > +static int allocate_first_block(int fd) > +{ > + ssize_t n; > + > + do { > + n = pwrite(fd, "\0", 1, 0); This breaks when fd has been opened with O_DIRECT.Quoted 1 line
It seems that we always open images without
O_DIRECTwhen creating animage
Quoted 1 line
in qemu-img create, or when creating a target image in qemu-img convert.
Yes. But you don’t call this function directly from image creation code but instead from the truncation function. (The former also calls the latter, but truncating is also an operation on its own.)
[...]
Quoted 4 lines
(Which happens when you open some file with cache.direct=on, and then use e.g. QMP’s
block_resize.)
Quoted 6 lines
Quoted 1 line
What would be a command triggering this? I can add a test.
block_resize, as I’ve said:$ ./qemu-img create -f raw empty.img 0
This is extreme edge case - why would someone create such image?
Quoted 6 lines
$ x86_64-softmmu/qemu-system-x86_64 \
-qmp stdio \
-blockdev file,node-name=file,filename=empty.img,cache.direct=on \
<<EOF
{'execute':'qmp_capabilities'}This is probably too late for the allocation, since we already probed the alignment before executing block_resize, and used a safe fallback (4096).
It can help if the image is reopened, since we probe alignment again.
Quoted 3 lines
{'execute':'block_resize',
'arguments':{'node-name':'file','size':1048576}}
EOF
Quoted 4 lines
$ ./qemu-img map empty.img Offset Length Mapped to File(You’d expect a data chunk here.)
I suppose you can get the same effect with blockdev-create and some
Quoted 3 lines
format that explicitly resizes the file to some target length (LUKS does this, I think), but this is the most direct route.
I will try to handle -blockdev in the next version.
Quoted 52 lines
It isn’t that bad because eventually you simply ignore the error. But
Quoted 2 lines
it still makes me wonder whether we shouldn’t write like the biggest power of two that does not exceed the new file length or
MAX_BLOCKSIZE.Quoted 5 lines
It makes sense if there is a way to cause qemu-img to use
O_DIRECTwhen creating an image.Quoted 9 lines
block/file-posix.c static int handle_aiocb_truncate(void
Quoted 9 lines
block/file-posix.c +5
-0+ }while (n ==-1 && errno ==EINTR ); + +return (n ==-1 ) ? -errno :0 ; +} +static int handle_aiocb_truncate (void *opaque) {RawPosixAIOData *aiocb = opaque;@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void*opaque)
Quoted 3 lines
block/file-posix.c static int handle_aiocb_truncate(void *opaque)
Quoted 3 lines
block/file-posix.c +0
-0/* posix_fallocate() doesn't set errno. */ error_setg_errno(errp, -result, "Could not preallocate newdata");
Quoted 19 lines
block/file-posix.c static int handle_aiocb_truncate(void *opaque) · line 1819
Quoted 2 lines
block/file-posix.c +2
-0+ }else if (current_length ==0 ) { +allocate_first_block (fd);Should
posix_fallocate()not take care of precisely this?Only if the filesystem does not support
fallocate()(e.g. NFS < 4.2).In this case
posix_fallocate()is doing:for (offset += (len - 1) % increment; len > 0; offset += increment) { len -= increment; if (offset < st.st_size) { unsigned char c; ssize_t rsize = __pread (fd, &c, 1, offset); if (rsize < 0) return errno;Quoted 10 lines
/* If there is a non-zero byte, the block must have been allocated already. */ else if (rsize == 1 && c != 0) continue; } if (__pwrite (fd, "", 1, offset) != 1) return errno; }
Quoted 8 lines
https://code.woboq.org/userspace/glibc/sysdeps/posix/posix_fallocate.c.html#96
Quoted 3 lines
So opening a file with
O_DIRECTwill break preallocation=falloc on such filesystems,But won’t the function above just fail with EINVAL?
allocate_first_block()is executed only in case of success.
Sure, but if posix_fallocate() fails, we fail qemu-img create/convert.
Quoted 30 lines
and writing one byte in
allocate_first_block()is safe.Quoted 5 lines
block/file-posix.c static int handle_aiocb_truncate(void *opaque) · line 1822
Quoted 4 lines
block/file-posix.c +0
-0} }else { result =0 ;Quoted 16 lines
[...] > diff --git a/tests/qemu-iotests/160 b/tests/qemu-iotests/160 > index df89d3864b..ad2d054a47 100755 > --- a/tests/qemu-iotests/160 > +++ b/tests/qemu-iotests/160 > @@ -57,6 +57,10 @@ for skip in $TEST_SKIP_BLOCKS; do > $QEMU_IMG dd if="$TEST_IMG" of="$TEST_IMG.out" skip="$skip" -O "$IMGFMT" \ > 2> /dev/null > TEST_IMG="$TEST_IMG.out" _check_test_img > + > + # We always write the first byte of an image. > + printf "\0" > "$TEST_IMG.out.dd" > + > dd if="$TEST_IMG" of="$TEST_IMG.out.dd" skip="$skip"status=none
Quoted 6 lines
Won’t this dd completely overwrite $TEST_IMG.out.dd (especially given the lack of conv=notrunc)?
There is an issue only if dd open the file with
O_TRUNC.
Quoted 8 lines
It isn’t an issue, I just don’t understand why the printf would be necessary at all.
dd should always truncate the output image unless conv=notrunc is specified. But even if it didn’t do that, in all of these test cases it should copy some data from $TEST_IMG to the output, and thus should always overwrite the first byte anyway.
Right, this change is not needed.
Quoted 35 lines
I will test
Quoted 10 lines
this again.
tests/qemu-iotests/175 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 37
Quoted 8 lines
echo
tests/qemu-iotests/175 +0
-0@@ -37,14 +37,16 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # the file size. This function hides the resulting difference inthe
Quoted 23 lines
tests/qemu-iotests/175 trap "_cleanup; exit \$status" 0 1 2 3 15
Quoted 13 lines
tests/qemu-iotests/175 +4
-3# stat -c '%b' output. # Parameter 1: Number of blocks an empty file occupies -# Parameter 2: Image size in bytes +# Parameter 2: Minimal number of blocks in an image +# Parameter 3: Image size in bytes _filter_blocks() { extra_blocks=$1 - img_size=$2 + min_blocks=$2 + img_size=$3 - sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothingallocated/" \ > - -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/" > + sed -e "s/blocks=$((extra_blocks + min_blocks))\\(\$\\|[^0-9]\\)/min allocation/" \ I don’t think adding extra_blocks to min_blocks makes sense. Just min_blocks alone should be what we want here.
Quoted 8 lines
Quoted 3 lines
We had failing tests (in vdsm) showing that filesystem may return more blocs than
expected even for non-empty files, so this may be needed.But
min_blocksis exactly the number of blocks of a file that has one allocated block. I don’t see how adding the number of blocks an empty file occupies makes sense.
You are right. The test fails on filesystem that allocates extra blocks.
Nir
Quoted 138 lines · Nir Soffer
On Thu, Aug 22, 2019 at 9:11 PM Max Reitz <[email protected] <mailto:[email protected]>> wrote:
On 22.08.19 18:39, Nir Soffer wrote:
> On Thu, Aug 22, 2019 at 5:28 PM Max Reitz <[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> On 16.08.19 23:21, Nir Soffer wrote:
> > When creating an image with preallocation "off" or "falloc", the first
> > block of the image is typically not allocated. When using Gluster
> > storage backed by XFS filesystem, reading this block using direct I/O
> > succeeds regardless of request length, fooling alignment detection.
> >
> > In this case we fallback to a safe value (4096) instead of the optimal
> > value (512), which may lead to unneeded data copying when aligning
> > requests. Allocating the first block avoids the fallback. > >
> > When using preallocation=off, we always allocate at least one > filesystem
> > block:
> >
> > $ ./qemu-img create -f raw test.raw 1g
> > Formatting 'test.raw', fmt=raw size=1073741824
> >
> > $ ls -lhs test.raw
> > 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
> >
> > I did quick performance tests for these flows:
> > - Provisioning a VM with a new raw image.
> > - Copying disks with qemu-img convert to new raw target image > >
> > I installed Fedora 29 server on raw sparse image, measuring the time
> > from clicking "Begin installation" until the "Reboot" button appears:
> >
> > Before(s) After(s) Diff(%)
> > -------------------------------
> > 356 389 +8.4
> >
> > I ran this only once, so we cannot tell much from these results. >
> So you’d expect it to be fast but it was slower? Well, you only ran it
> once and it isn’t really a precise benchmark...
>
> > The second test was cloning the installation image with qemu-img > > convert, doing 10 runs:
> >
> > for i in $(seq 10); do
> > rm -f dst.raw
> > sleep 10
> > time ./qemu-img convert -f raw -O raw -t none -T none > src.raw dst.raw
> > done
> >
> > Here is a table comparing the total time spent:
> >
> > Type Before(s) After(s) Diff(%)
> > ---------------------------------------
> > real 530.028 469.123 -11.4
> > user 17.204 10.768 -37.4
> > sys 17.881 7.011 -60.7
> >
> > Here we see very clear improvement in CPU usage.
> >
> > Signed-off-by: Nir Soffer <[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> > > ---
> > block/file-posix.c | 25 +++++++++++++++++++++++++ > > tests/qemu-iotests/150.out | 1 +
> > tests/qemu-iotests/160 | 4 ++++
> > tests/qemu-iotests/175 | 19 +++++++++++++------ > > tests/qemu-iotests/175.out | 8 ++++----
> > tests/qemu-iotests/221.out | 12 ++++++++----
> > tests/qemu-iotests/253.out | 12 ++++++++----
> > 7 files changed, 63 insertions(+), 18 deletions(-) > >
> > diff --gita/block/file-posix.cb/block/file-posix.c> > indexb9c33c8f6c..3964dd2021100644
> > ---a/block/file-posix.c
> > +++b/block/file-posix.c
> > @@ -1755,6 +1755,27 @@ static inthandle_aiocb_discard(void *opaque)
> > return ret;
> > }
> >
> > +/*
> > + * Help alignment detection by allocating the first block. > > + *
> > + * When reading with direct I/O from unallocated area on Gluster
> backed by XFS,
> > + * reading succeeds regardless of request length. In this case we
> fallback to
> > + * safe aligment which is not optimal. Allocating the first block
> avoids this
> > + * fallback.
> > + *
> > + * Returns: 0 on success, -errno on failure.
> > + */
> > +static intallocate_first_block(int fd)
> > +{
> > +ssize_tn;
> > +
> > + do {
> > + n =pwrite(fd, "\0", 1, 0);
>
> This breaks when fd has been opened withO_DIRECT.
>
>
> It seems that we always open images withoutO_DIRECTwhen creating an image
> in qemu-img create, or when creating a target image in qemu-img convert.Yes. But you don’t call this function directly from image creation code but instead from the truncation function. (The former also calls the latter, but truncating is also an operation on its own.)
[...]
> (Which happens when you open some file with cache.direct=on, and then
> use e.g. QMP’sblock_resize.)
>
>
Quoted 8 lines
> What would be a command triggering this? I can add a test.
block_resize, as I’ve said:$ ./qemu-img create -f raw empty.img 0
This is extreme edge case - why would someone create such image?
Because it works?
This is generally the fist step of image creation with blockdev-create, because you don’t care about the size of the protocol layer.
If you have a format layer that truncates the image to a fixed size and does not write anything into the first block itself (say because it uses a footer), then (with O_DIRECT) allocate_first_block() will fail (silently, because while it does return an error value, it is never checked and there is no comment that explains why we don’t check it) and the first block actually will not be allocated.
I could show you that with VPC (which supports a fixed subformat where it uses a footer), but unfortunately that’s a bit broken right now (because of a bug in blockdev-create; I’ll send a patch).
The test would go like this:
$ x86_64-softmmu/qemu-system-x86_64 -qmp stdio
{"execute":"qmp_capabilities"}{"execute":"blockdev-create",
"arguments":{
"job-id":"create",
"options":{"driver":"file",
"filename":"test.img",
"size":0}}}[Wait until the job is pending]
{"execute":"job-dismiss","arguments":{"id":"create"}}
{"execute":"blockdev-add",
"arguments":{
"driver":"file",
"node-name":"protocol-node",
"filename":"test.img",
"cache":{"direct":true}}}{"execute":"blockdev-create",
"arguments":{
"job-id":"create",
"options":{"driver":"vpc",
"file":"protocol-node",
"subformat":"fixed",
"size":67108864,
"force-size":true}}}[Wait until the job is pending]
{"execute":"job-dismiss","arguments":{"id":"create"}}
{"execute":"quit"}
And then:
$ ./qemu-img map test.img Offset Length Mapped to File 0x4000000 0x200 0x4000000 test.img
The footer is mapped, but the first block is not allocated.
As I said, for that to work, you need a patch (because of a bug), namely:
[Start of patch]
block/create.c +0 -0
@@ -48,7 +48,7 @@ static int coroutine_fn blockdev_create_run(Job *job,
Error **errp)
qapi_free_BlockdevCreateOptions(s->opts);
- return ret;
+ return ret < 0 ? ret : 0;
}
static const JobDriver blockdev_create_job_driver = {
[End of patch]
(The reason being that the vpc block driver returns 512 here to signify success, but the job infrastructure treats anything but 0 as a failure.)
Quoted 11 lines
$ x86_64-softmmu/qemu-system-x86_64 \ -qmp stdio \ -blockdev file,node-name=file,filename=empty.img,cache.direct=on \ <<EOF {'execute':'qmp_capabilities'}This is probably too late for the allocation, since we already probed the alignment before executing
block_resize, and used a safe fallback (4096).
It can help if the image is reopened, since we probe alignment again.
I’m not talking about getting the alignment right when you have a zero-length image. That can probably never work with probing. (Well, I mean, technically you could make allocate_first_block() probe. I won’t ask for that because that really seems like too little gain for too much effort.)
I’m just talking about the fact that this allocating write will fail, so when the image is used the next time, it will not have the first block allocated.
[...]
Quoted 45 lines
Quoted 1 line
> @@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void
*opaque) > > /* posix_fallocate() doesn't set errno. */ > > error_setg_errno(errp, -result, > > "Could not preallocate new data"); > > + } else if (current_length == 0) { > > + allocate_first_block(fd); > > Should posix_fallocate() not take care of precisely this? > > > Only if the filesystem does not support fallocate() (e.g. NFS < 4.2). > > In this case posix_fallocate() is doing: > > for (offset += (len - 1) % increment; len > 0; offset += increment) > { > len -= increment; > if (offset < st.st_size) > { > unsigned char c; > ssize_t rsize = __pread (fd, &c, 1, offset); > if (rsize < 0) > return errno; > /* If there is a non-zero byte, the block must have been > allocated already. */ > else if (rsize == 1 && c != 0) > continue; > } > if (__pwrite (fd, "", 1, offset) != 1) > return errno; > } > > https://code.woboq.org/userspace/glibc/sysdeps/posix/posix_fallocate.c.html#96 > > So opening a file with O_DIRECT will break preallocation=falloc on such > filesystems, But won’t the function above just fail with EINVAL? allocate_first_block() is executed only in case of success.
Quoted 1 line
Sure, but if
posix_fallocate()fails, we fail qemu-img create/convert.
Exactly. But if posix_fallocate() works, it should have allocated the first block.
Max
Quoted 2 lines · Max Reitz
On 22.08.19 21:01, Nir Soffer wrote:
...
Quoted 44 lines
Quoted 13 lines
Quoted 1 line
Review static int handle_aiocb_truncate(void
Quoted 1 line
(quoted diff) +0
-0@@ -1794,6 +1815,8 @@ static int handle_aiocb_truncate(void*opaque) > > /* posix_fallocate() doesn't set errno. */ > > error_setg_errno(errp, -result, > > "Could not preallocate new data"); > > + } else if (current_length == 0) { > > + allocate_first_block(fd); > > Should posix_fallocate() not take care of precisely this? > > > Only if the filesystem does not support fallocate() (e.g. NFS <4.2).
Quoted 4 lines
Quoted 4 lines
In this case
posix_fallocate()is doing:for (offset += (len - 1) % increment; len > 0; offset +=
increment)
Quoted 10 lines
Quoted 10 lines
{ len -= increment; if (offset < st.st_size) { unsigned char c; ssize_t rsize = __pread (fd, &c, 1, offset); if (rsize < 0) return errno; /* If there is a non-zero byte, the block must have been allocated already. */Quoted 9 lines
Quoted 8 lines
else if (rsize == 1 && c != 0) continue; } if (__pwrite (fd, "", 1, offset) != 1) return errno; }https://code.woboq.org/userspace/glibc/sysdeps/posix/posix_fallocate.c.html#96
Quoted 5 lines
Quoted 2 lines
So opening a file with
O_DIRECTwill break preallocation=falloc onsuch
> filesystems,
Quoted 9 lines
Quoted 5 lines
But won’t the function above just fail with EINVAL?
allocate_first_block()is executed only in case of success.Sure, but if
posix_fallocate()fails, we fail qemu-img create/convert.Exactly. But if
posix_fallocate()works, it should have allocated the first block.
Only if the file system does not support fallocate(). posix_fallocate() first tryfallocate(), and fall back to manual preallocation:
https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/posix_fallocate.c.html#27
Here is an example using fallocate --posix:
$ sudo mount -t glusterfs gluster1:/gv0 /tmp/gv0
(gv0 is gluster volume backed by XFS on top of VDO device with 4k sector size)
$ fallocate -l 1g --posix empty.raw $ dd if=empty.raw bs=1 count=1 of=/dev/null iflag=direct status=none
$ dd if=/dev/zero bs=1 count=1 of=empty.raw conv=notrunc status=none $ dd if=empty.raw bs=1 count=1 of=/dev/null iflag=direct status=none dd: error reading 'empty.raw': Invalid argument
$ dd if=empty.raw bs=512 count=1 of=/dev/null iflag=direct status=none dd: error reading 'empty.raw': Invalid argument
$ dd if=empty.raw bs=4096 count=1 of=/dev/null iflag=direct status=none
Here is example using gluster storage with sector size of 512 bytes.
$ sudo mount -t glusterfs gluster1:/gv1 /tmp/gv1
$ fallocate -l 1g --posix empty.raw $ dd if=empty.raw bs=1 count=1 of=/dev/null iflag=direct status=none
$ dd if=/dev/zero bs=1 count=1 of=empty.raw conv=notrunc status=none
$ dd if=empty.raw bs=1 count=1 of=/dev/null iflag=direct status=none dd: error reading 'empty.raw': Invalid argument
$ dd if=empty.raw bs=512 count=1 of=/dev/null iflag=direct status=none
So we must allocated using write() after calling posix_fallocate().
Nir
Quoted 3 lines · Nir Soffer
On Fri, Aug 23, 2019 at 4:58 PM Max Reitz <[email protected] <mailto:[email protected]>> wrote:
[...]
Quoted 8 lines
Exactly. But if
posix_fallocate()works, it should have allocated the first block.Only if the file system does not support
fallocate().posix_fallocate()first tryfallocate(), and fall back to manual preallocation:
https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/posix_fallocate.c.html#27
I still don’t understand. Your example does show that the first block is not allocated by fallocate(), but I still don’t understand the connection to not having fallocate() support.
If it doesn’t have fallocate() support and posix_fallocate() does fall back, the result should be that posix_fallocate() manually allocates data, which should be completely sufficient.
So in fact, it seems to me that the opposite is true: It seems that when allocating blocks on XFS with fallocate(), that simply won’t be enough to cause alignment errors. So it doesn’t seem to be about fallback code, but precisely the normal XFS code that fully supports fallocate.
(Just running your example on a local file on XFS shows the same result.)
So that seems to me why the additional allocation is necessary. I think that should be noted in a comment – if I’m right (I may well not be).
Max
Quoted 153 lines · Max Reitz
Quoted 44 lines · Nir Soffer
On Thu, Aug 22, 2019 at 9:11 PM Max Reitz <[email protected] <mailto:[email protected]>> wrote:
On 22.08.19 18:39, Nir Soffer wrote:
> On Thu, Aug 22, 2019 at 5:28 PM Max Reitz <[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> On 16.08.19 23:21, Nir Soffer wrote:
> > When creating an image with preallocation "off" or "falloc",
the first
> > block of the image is typically not allocated. When using
Gluster
> > storage backed by XFS filesystem, reading this block using
direct I/O
> > succeeds regardless of request length, fooling alignment
detection.
> >
> > In this case we fallback to a safe value (4096) instead of
the optimal
> > value (512), which may lead to unneeded data copying when
aligning
> > requests. Allocating the first block avoids the fallback.
> >
> > When using preallocation=off, we always allocate at least one
> filesystem
> > block:
> >
> > $ ./qemu-img create -f raw test.raw 1g
> > Formatting 'test.raw', fmt=raw size=1073741824
> >
> > $ ls -lhs test.raw
> > 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48
test.raw
> >
> > I did quick performance tests for these flows:
> > - Provisioning a VM with a new raw image.
> > - Copying disks with qemu-img convert to new raw target image
> >
> > I installed Fedora 29 server on raw sparse image, measuring
the time
> > from clicking "Begin installation" until the "Reboot" button
appears:
> >Quoted 5 lines
> > Before(s) After(s) Diff(%)
> > -------------------------------
> > 356 389 +8.4
> >
> > I ran this only once, so we cannot tell much from theseresults.
Quoted 6 lines
Quoted 2 lines
So you’d expect it to be fast but it was slower? Well, you
only ran it
> once and it isn’t really a precise benchmark...
>
> > The second test was cloning the installation image withqemu-img
Quoted 65 lines
Quoted 6 lines
Quoted 6 lines
convert, doing 10 runs:
for i in $(seq 10); do
rm -f dst.raw
sleep 10
time ./qemu-img convert -f raw -O raw -t none -T noneQuoted 14 lines
src.raw dst.raw > done > > Here is a table comparing the total time spent: > > Type Before(s) After(s) Diff(%) > --------------------------------------- > real 530.028 469.123 -11.4 > user 17.204 10.768 -37.4 > sys 17.881 7.011 -60.7 > > Here we see very clear improvement in CPU usage. > > Signed-off-by: Nir Soffer <[email protected]<mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> > > --- > > block/file-posix.c | 25 +++++++++++++++++++++++++ > > tests/qemu-iotests/150.out | 1 + > > tests/qemu-iotests/160 | 4 ++++ > > tests/qemu-iotests/175 | 19 +++++++++++++------ > > tests/qemu-iotests/175.out | 8 ++++---- > > tests/qemu-iotests/221.out | 12 ++++++++---- > > tests/qemu-iotests/253.out | 12 ++++++++---- > > 7 files changed, 63 insertions(+), 18 deletions(-) > > > > diff --git a/block/file-posix.c b/block/file-posix.c > > index b9c33c8f6c..3964dd2021 100644 > > --- a/block/file-posix.c > > +++ b/block/file-posix.c > > @@ -1755,6 +1755,27 @@ static int handle_aiocb_discard(void *opaque) > > return ret; > > } > > > > +/* > > + * Help alignment detection by allocating the first block. > > + * > > + * When reading with direct I/O from unallocated area on Gluster > backed by XFS, > > + * reading succeeds regardless of request length. In this case we > fallback to > > + * safe aligment which is not optimal. Allocating the first block > avoids this > > + * fallback. > > + * > > + * Returns: 0 on success, -errno on failure. > > + */ > > +static int allocate_first_block(int fd) > > +{ > > + ssize_t n; > > + > > + do { > > + n = pwrite(fd, "\0", 1, 0); > > This breaks when fd has been opened with O_DIRECT.Quoted 8 lines
> > > It seems that we always open images without O_DIRECT when creating an image > in qemu-img create, or when creating a target image in qemu-img convert. Yes. But you don’t call this function directly from image creationcode
Quoted 10 lines
but instead from the truncation function. (The former also calls the latter, but truncating is also an operation on its own.)
[...]
> (Which happens when you open some file with cache.direct=on,
and then
> use e.g. QMP’sblock_resize.)
>
>Quoted 8 lines
> What would be a command triggering this? I can add a test.
block_resize, as I’ve said:$ ./qemu-img create -f raw empty.img 0
This is extreme edge case - why would someone create such image?
Because it works?
Quoted 8 lines
This is generally the fist step of image creation with blockdev-create, because you don’t care about the size of the protocol layer.
If you have a format layer that truncates the image to a fixed size and does not write anything into the first block itself (say because it uses a footer), then (with
O_DIRECT)allocate_first_block()will fail (silently, because while it does return an error value, it is never checked and there is no comment that explains why we don’t check it)
The motivation is that this is an optimization for the special case of using empty image, so it does not worth failing image creation.
I will add a comment about that.
Quoted 45 lines
and
the first block actually will not be allocated.I could show you that with VPC (which supports a fixed subformat where it uses a footer), but unfortunately that’s a bit broken right now (because of a bug in blockdev-create; I’ll send a patch).
The test would go like this:
$ x86_64-softmmu/qemu-system-x86_64 -qmp stdio {"execute":"qmp_capabilities"}{"execute":"blockdev-create",
"arguments":{ "job-id":"create", "options":{"driver":"file", "filename":"test.img", "size":0}}}[Wait until the job is pending]
{"execute":"job-dismiss","arguments":{"id":"create"}}
{"execute":"blockdev-add",
"arguments":{ "driver":"file", "node-name":"protocol-node", "filename":"test.img", "cache":{"direct":true}}}{"execute":"blockdev-create",
"arguments":{ "job-id":"create", "options":{"driver":"vpc", "file":"protocol-node", "subformat":"fixed", "size":67108864, "force-size":true}}}[Wait until the job is pending]
{"execute":"job-dismiss","arguments":{"id":"create"}}
{"execute":"quit"}
Quoted 8 lines
And then:
$ ./qemu-img map test.img Offset Length Mapped to File 0x4000000 0x200 0x4000000 test.imgThe footer is mapped, but the first block is not allocated.
Thanks for the example.
I will need time to play with blockdev and understand the flows when image are created. Do you think is would be useful to fix now only image creation via qemu-img, and handle blockdev later?
...
Quoted 2 lines · Nir Soffer
On Fri, Aug 23, 2019 at 4:58 PM Max Reitz <[email protected] <mailto:[email protected]>> wrote:
[...]
Quoted 10 lines
If you have a format layer that truncates the image to a fixed size and does not write anything into the first block itself (say because it uses a footer), then (with
O_DIRECT)allocate_first_block()will fail (silently, because while it does return an error value, it is never checked and there is no comment that explains why we don’t check it)The motivation is that this is an optimization for the special case of using empty image, so it does not worth failing image creation.
I will add a comment about that.
Thanks!
[...]
Quoted 5 lines
Thanks for the example.
I will need time to play with blockdev and understand the flows when image are created. Do you think is would be useful to fix now only image creation via qemu-img, and handle blockdev later?
Well, it isn’t about blockdev, it’s simply about the fact that this function doesn’t work for O_DIRECT files. I showed how to reproduce the issue without blockdev (namely block_resize). Sure, that is an edge case, but it is a completely valid case.
Also, it seems to me the fix is rather simple. Just something like:
static int allocate_first_block(int fd, int64_t max_size)
{
int write_size = MIN(max_size, MAX_BLOCKSIZE);
void *buf;
ssize_t n;
/* Round down to power of two */
assert(write_size > 0);
write_size = 1 << (31 - clz32(write_size));
buf = qemu_memalign(MAX(getpagesize(), write_size), write_size);
memset(buf, 0, write_size);
do {
n = pwrite(fd, buf, write_size, 0);
} while (n < 0 && errno == EINTR);
qemu_vfree(buf);
return n < 0 ? -errno : 0;
}Wouldn’t that work?
Max
Quoted 61 lines · Max Reitz
Quoted 2 lines · Nir Soffer
On Fri, Aug 23, 2019 at 4:58 PM Max Reitz <[email protected] <mailto:[email protected]>> wrote:
[...]
Quoted 1 line
If you have a format layer that truncates the image to a fixed size
and
Quoted 1 line
does not write anything into the first block itself (say because it
uses
Quoted 6 lines
a footer), then (with
O_DIRECT)allocate_first_block()will fail (silently, because while it does return an error value, it is never checked and there is no comment that explains why we don’t check it)The motivation is that this is an optimization for the special case of
using
Quoted 2 lines
empty image, so it does not worth failing image creation. I will add a comment about that.
Thanks!
[...]
Quoted 3 lines
Thanks for the example.
I will need time to play with blockdev and understand the flows when
image
Quoted 1 line
are created. Do you think is would be useful to fix now only image
creation
Quoted 1 line
via qemu-img, and handle blockdev later?
Well, it isn’t about blockdev, it’s simply about the fact that this function doesn’t work for
O_DIRECTfiles. I showed how to reproduce the issue without blockdev (namelyblock_resize). Sure, that is an edge case, but it is a completely valid case.Also, it seems to me the fix is rather simple. Just something like:
static int
allocate_first_block(int fd, int64_t max_size){ int write_size = MIN(max_size, MAX_BLOCKSIZE); void *buf; ssize_t n; /* Round down to power of two */ assert(write_size > 0); write_size = 1 << (31 - clz32(write_size)); buf = qemu_memalign(MAX(getpagesize(), write_size), write_size); memset(buf, 0, write_size); do { n = pwrite(fd, buf, write_size, 0); } while (n < 0 && errno == EINTR); qemu_vfree(buf); return n < 0 ? -errno : 0; }
Quoted 2 lines
Wouldn’t that work?
Sure, it should work.
But I think we can make this simpler, always writing MIN(max_size, MAX_BLOCKSIZE).
vdsm is enforcing now 4k alignment, and there is no way to create images with unaligned
size. Maybe qemu should adapt this rule?
Nir
Quoted 9 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
Quoted 8 lines
When using preallocation=off, we always allocate at least one filesystem block:
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
Are you sure about this?
[mlevitsk@maximlenovopc ~/work/test_area/posix-file 0]$ qemu-img create -f raw test.raw 1g -o preallocation=off Formatting 'test.raw', fmt=raw size=1073741824 preallocation=off
[mlevitsk@maximlenovopc ~/work/test_area/posix-file 0]$ls -lhs ./test.raw
0 -rw-r--r--. 1 mlevitsk mlevitsk 1.0G Aug 25 10:38 ./test.raw
ext4, tested on qemu-4.0.0 and qemu git master.
From what I remember, the only case when posix-raw touches the first block is to zero it out when running on top of kernel block device, to erase whatever header might be there, and this is also kind of a backward compat hack which might be one day removed.
[...]
Best regards,
Maxim Levitsky
Quoted 13 lines · Maxim Levitsky
Quoted 12 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
When using preallocation=off, we always allocate at least one filesystem block:
Quoted 8 lines
Quoted 5 lines
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawAre you sure about this?
This is the new behaviour with this change...
[mlevitsk@maximlenovopc ~/work/test_area/posix-file 0]$ qemu-img create -f
Quoted 7 lines
raw test.raw 1g -o preallocation=off
Formatting 'test.raw', fmt=raw size=1073741824 preallocation=off [mlevitsk@maximlenovopc ~/work/test_area/posix-file 0]$ls -lhs ./test.raw 0 -rw-r--r--. 1 mlevitsk mlevitsk 1.0G Aug 25 10:38 ./test.rawext4, tested on qemu-4.0.0 and qemu git master.
And this is the old behavior. I guess the commit message does not make it clear.
From what I remember, the only case when posix-raw touches the first block
Quoted 5 lines
is to zero it out
when running on top of kernel block device, to erase whatever header might be there, and this
is also kind of a backward compat hack which might be one day removed.
This change is only for file, on block storage we use BLKSSZGET.
Quoted 8 lines
[...]
Best regards,
Maxim Levitsky
Quoted 24 lines · Nir Soffer
Quoted 13 lines · Maxim Levitsky
Quoted 12 lines · Nir Soffer
When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection.
In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback.
When using preallocation=off, we always allocate at least one filesystem block:
Quoted 7 lines
Quoted 5 lines
$ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824$ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.rawAre you sure about this?
This is the new behaviour with this change...
Quoted 8 lines
Quoted 6 lines
[mlevitsk@maximlenovopc ~/work/test_area/posix-file 0]$ qemu-img create -f raw test.raw 1g -o preallocation=off Formatting 'test.raw', fmt=raw size=1073741824 preallocation=off
[mlevitsk@maximlenovopc ~/work/test_area/posix-file 0]$ls -lhs ./test.raw
0 -rw-r--r--. 1 mlevitsk mlevitsk 1.0G Aug 25 10:38 ./test.rawext4, tested on qemu-4.0.0 and qemu git master.
And this is the old behavior. I guess the commit message does not make it clear.
Ah, thanks!
Quoted 4 lines
Quoted 3 lines
From what I remember, the only case when posix-raw touches the first block is to zero it out when running on top of kernel block device, to erase whatever header might be there, and this is also kind of a backward compat hack which might be one day removed.
Quoted 8 lines
This change is only for file, on block storage we use BLKSSZGET.
Quoted 6 lines
[...]
Best regards,
Maxim Levitsky
Best regards,
Maxim Levitsky