Skip to contentSkip to conversation

Block layer patches

GIT PULL New Kevin Wolf 3 participants 9 messages Jul 9, 2020 – Jul 13, 2020

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

The following changes since commit 8796c64ecdfd34be394ea277aaaaa53df0c76996:

Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 16:33:59 +0100)

are available in the Git repository at:

git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to d1c824e681e423a6224c6831a493e9175fa02dc1:

qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200)

----------------------------------------------------------------
Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size

----------------------------------------------------------------
Eric Blake (10):
qemu-img: Flush stdout before before potential stderr messages
block: Finish deprecation of 'qemu-img convert -n -o'
sheepdog: Add trivial backing_fmt support
vmdk: Add trivial backing_fmt support
qcow: Tolerate backing_fmt=
block: Error if backing file fails during creation without -u
qcow2: Deprecate use of qemu-img amend to change backing file
iotests: Specify explicit backing format where sensible
block: Add support to warn on backing file change without format
qemu-img: Deprecate use of -b without -F

Kevin Wolf (2):
qemu-img map: Don't limit block status request size
file-posix: Mitigate file fragmentation with extent size hints

Max Reitz (1):
iotests: Simplify _filter_img_create() a bit

 qapi/block-core.json             | 11 +++--
 docs/system/deprecated.rst       | 58 ++++++++++++++++++++++----
 docs/tools/qemu-img.rst          |  4 ++
 include/block/block.h            |  4 +-
 include/block/block_int.h        |  1 +
 block.c                          | 53 +++++++++++++++++-------
 block/file-posix.c               | 42 +++++++++++++++++++
 block/qcow.c                     | 20 ++++++++-
 block/qcow2.c                    |  7 +++-
 block/sheepdog.c                 | 18 +++++++-
 block/stream.c                   |  2 +-
 block/vmdk.c                     | 14 +++++++
 blockdev.c                       |  3 +-
 qemu-img.c                       | 20 +++++----
 tests/qemu-iotests/017           |  2 +-
 tests/qemu-iotests/017.out       |  2 +-
 tests/qemu-iotests/018           |  2 +-
 tests/qemu-iotests/018.out       |  2 +-
 tests/qemu-iotests/019           |  5 ++-
 tests/qemu-iotests/019.out       |  2 +-
 tests/qemu-iotests/020           | 31 ++++++++------
 tests/qemu-iotests/020.out       | 15 +------
 tests/qemu-iotests/024           |  8 ++--
 tests/qemu-iotests/024.out       |  5 ++-
 tests/qemu-iotests/028           |  4 +-
 tests/qemu-iotests/028.out       |  2 +-
 tests/qemu-iotests/030           | 26 ++++++++----
 tests/qemu-iotests/034           |  2 +-
 tests/qemu-iotests/034.out       |  2 +-
 tests/qemu-iotests/037           |  2 +-
 tests/qemu-iotests/037.out       |  2 +-
 tests/qemu-iotests/038           |  2 +-
 tests/qemu-iotests/038.out       |  2 +-
 tests/qemu-iotests/039           |  3 +-
 tests/qemu-iotests/039.out       |  2 +-
 tests/qemu-iotests/040           | 47 +++++++++++++++------
 tests/qemu-iotests/041           | 37 +++++++++++------
 tests/qemu-iotests/042           |  4 +-
 tests/qemu-iotests/043           | 18 ++++----
 tests/qemu-iotests/043.out       | 16 +++++---
 tests/qemu-iotests/046           |  2 +-
 tests/qemu-iotests/046.out       |  2 +-
 tests/qemu-iotests/049.out       |  8 ++--
 tests/qemu-iotests/050           |  4 +-
 tests/qemu-iotests/050.out       |  2 +-
 tests/qemu-iotests/051           |  2 +-
 tests/qemu-iotests/051.out       |  2 +-
 tests/qemu-iotests/051.pc.out    |  2 +-
 tests/qemu-iotests/054.out       |  2 +-
 tests/qemu-iotests/056           |  3 +-
 tests/qemu-iotests/060           |  2 +-
 tests/qemu-iotests/060.out       |  2 +-
 tests/qemu-iotests/061           | 10 ++---
 tests/qemu-iotests/061.out       | 11 ++---
 tests/qemu-iotests/069           |  2 +-
 tests/qemu-iotests/069.out       |  2 +-
 tests/qemu-iotests/073           |  2 +-
 tests/qemu-iotests/073.out       |  2 +-
 tests/qemu-iotests/079.out       |  2 +-
 tests/qemu-iotests/082           | 10 +++--
 tests/qemu-iotests/082.out       | 30 +++++++++++---
 tests/qemu-iotests/085           |  4 +-
 tests/qemu-iotests/085.out       |  6 +--
 tests/qemu-iotests/089           |  2 +-
 tests/qemu-iotests/089.out       |  2 +-
 tests/qemu-iotests/095           |  4 +-
 tests/qemu-iotests/095.out       |  4 +-
 tests/qemu-iotests/097           |  4 +-
 tests/qemu-iotests/097.out       | 16 ++++----
 tests/qemu-iotests/098           |  2 +-
 tests/qemu-iotests/098.out       |  8 ++--
 tests/qemu-iotests/110           |  4 +-
 tests/qemu-iotests/110.out       |  4 +-
 tests/qemu-iotests/111.out       |  2 +-
 tests/qemu-iotests/112.out       |  4 +-
 tests/qemu-iotests/114           | 12 ++++++
 tests/qemu-iotests/114.out       |  9 ++++
 tests/qemu-iotests/122           | 34 +++++++++++-----
 tests/qemu-iotests/122.out       | 12 ++++--
 tests/qemu-iotests/126           |  4 +-
 tests/qemu-iotests/126.out       |  4 +-
 tests/qemu-iotests/127           |  4 +-
 tests/qemu-iotests/127.out       |  4 +-
 tests/qemu-iotests/129           |  3 +-
 tests/qemu-iotests/133           |  2 +-
 tests/qemu-iotests/133.out       |  2 +-
 tests/qemu-iotests/139           |  2 +-
 tests/qemu-iotests/141           |  6 +--
 tests/qemu-iotests/141.out       |  4 +-
 tests/qemu-iotests/142           |  2 +-
 tests/qemu-iotests/142.out       |  2 +-
 tests/qemu-iotests/153           | 14 +++----
 tests/qemu-iotests/153.out       | 35 ++++++++--------
 tests/qemu-iotests/154           | 42 +++++++++----------
 tests/qemu-iotests/154.out       | 42 +++++++++----------
 tests/qemu-iotests/155           | 12 ++++--
 tests/qemu-iotests/156           |  9 ++--
 tests/qemu-iotests/156.out       |  6 +--
 tests/qemu-iotests/158           |  2 +-
 tests/qemu-iotests/158.out       |  2 +-
 tests/qemu-iotests/161           |  8 ++--
 tests/qemu-iotests/161.out       |  8 ++--
 tests/qemu-iotests/176           |  4 +-
 tests/qemu-iotests/176.out       | 32 +++++++--------
 tests/qemu-iotests/177           |  2 +-
 tests/qemu-iotests/177.out       |  2 +-
 tests/qemu-iotests/179           |  2 +-
 tests/qemu-iotests/179.out       |  2 +-
 tests/qemu-iotests/189           |  2 +-
 tests/qemu-iotests/189.out       |  2 +-
 tests/qemu-iotests/191           | 12 +++---
 tests/qemu-iotests/191.out       | 12 +++---
 tests/qemu-iotests/195           |  6 +--
 tests/qemu-iotests/195.out       |  6 +--
 tests/qemu-iotests/198           |  2 +-
 tests/qemu-iotests/198.out       |  3 +-
 tests/qemu-iotests/204           |  2 +-
 tests/qemu-iotests/204.out       |  2 +-
 tests/qemu-iotests/216           |  2 +-
 tests/qemu-iotests/224           |  4 +-
 tests/qemu-iotests/225           |  2 +-
 tests/qemu-iotests/225.out       |  2 +-
 tests/qemu-iotests/228           |  5 ++-
 tests/qemu-iotests/243           |  7 ++--
 tests/qemu-iotests/245           |  3 +-
 tests/qemu-iotests/249           |  4 +-
 tests/qemu-iotests/249.out       |  4 +-
 tests/qemu-iotests/252           |  2 +-
 tests/qemu-iotests/257           |  3 +-
 tests/qemu-iotests/259.out       |  2 +-
 tests/qemu-iotests/267           |  4 +-
 tests/qemu-iotests/267.out       |  6 +--
 tests/qemu-iotests/270           |  2 +-
 tests/qemu-iotests/270.out       |  2 +-
 tests/qemu-iotests/273           |  4 +-
 tests/qemu-iotests/273.out       |  4 +-
 tests/qemu-iotests/274           | 12 +++---
 tests/qemu-iotests/274.out       | 29 +++++++------
 tests/qemu-iotests/279           |  4 +-
 tests/qemu-iotests/279.out       |  4 +-
 tests/qemu-iotests/290           |  2 +-
 tests/qemu-iotests/290.out       |  4 +-
 tests/qemu-iotests/301           | 88 ++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/301.out       | 59 +++++++++++++++++++++++++++
 tests/qemu-iotests/common.filter | 62 ++++++++++++----------------
 tests/qemu-iotests/group         |  1 +
 146 files changed, 906 insertions(+), 472 deletions(-)
 create mode 100755 tests/qemu-iotests/301
 create mode 100644 tests/qemu-iotests/301.out
#1 lore raw
Quoted 11 lines · Kevin Wolf

The following changes since commit 8796c64ecdfd34be394ea277aaaaa53df0c76996:

Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 16:33:59 +0100)

are available in the Git repository at:

git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to d1c824e681e423a6224c6831a493e9175fa02dc1:

Quoted 9 lines

qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200)

----------------------------------------------------------------
Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size

iotest 114 fails on FreeBSD and OpenBSD (and probably NetBSD but that build hasn't reported back yet): looks like a non-portable use of 'truncate' ?

TEST iotest-qcow2: 114 [fail]
QEMU --
"/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64" -nodefaults -disp
lay none -machine virt -accel qtest
QEMU_IMG --
"/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO --
"/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -f qcow2
QEMU_NBD --
"/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/../../qemu-nbd"

IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
TEST_DIR      -- /home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.RQQ2LyuG

SOCKET_SCM_HELPER --

--- /home/qemu/qemu-test.FOdDM2/src/tests/qemu-iotests/114.out

2020-07-10 14:51:21.000000000 +0000
+++ /home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/114.out.bad
 2020-07-10 15:15:14.446278000 +0000

@@ -1,8 +1,11 @@
QA output created by 114
-qemu-img: warning: Deprecated use of backing file without explicit
backing format (detected format of raw)
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.orig
+truncate: illegal option -- -
+usage: truncate [-c] -s [+|-|%|/]size[K|k|M|m|G|g|T|t] file ...
+ truncate [-c] -r rfile file ...
+qemu-img: TEST_DIR/t.IMGFMT: Could not open 'TEST_DIR/t.IMGFMT.orig':
No such file or directory
+Could not open backing image.
Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864
-qemu-img: warning: Deprecated use of backing file without explicit
backing format
+qemu-img: Could not open '/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/scratch/t.qcow2.orig':
Could not open '/home/qemu/qemu-test.FOdDM2/build/tests/qemu-iotests/scratch/t.qcow2.orig':
No such file or directory
qemu-img: warning: Deprecated use of backing file without explicit
backing format (detected format of IMGFMT)
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
qemu-img: warning: Deprecated use of unopened backing file without
explicit backing format, use of this image requires potentially unsafe format probing

thanks
-- PMM

#2 lore raw
Quoted 21 lines · Peter Maydell
Quoted 9 lines
   qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200)

----------------------------------------------------------------
Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size

iotest 114 fails on FreeBSD and OpenBSD (and probably NetBSD but that build hasn't reported back yet): looks like a non-portable use of 'truncate' ?

(quoted diff) +0 -2
@@ -1,8 +1,11 @@
  QA output created by 114
-qemu-img: warning: Deprecated use of backing file without explicit
 backing format (detected format of raw)
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864

backing_file=TEST_DIR/t.IMGFMT.orig

Review

Quoted 4 lines
(quoted diff) +4 -0
+truncate: illegal option -- -
+usage: truncate [-c] -s [+|-|%|/]size[K|k|M|m|G|g|T|t] file ...
+       truncate [-c] -r rfile file ...
+qemu-img: TEST_DIR/t.IMGFMT: Could not open 'TEST_DIR/t.IMGFMT.orig':

Yep, 'truncate' is not POSIX, but we've been lucky that most of our uses have been under '_supported_os Linux'; test 272 runs on BSD with 'truncate -s' instead of 'truncate --size'. The fix is obvious; I can post the followup to squash in if that will help.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw
In reply to Kevin Wolf
Quoted 19 lines · Kevin Wolf

The following changes since commit 8796c64ecdfd34be394ea277aaaaa53df0c76996:

   Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 16:33:59 +0100)

are available in the Git repository at:

git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to d1c824e681e423a6224c6831a493e9175fa02dc1:

   qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200)

----------------------------------------------------------------
Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size

Quoted 7 lines

----------------------------------------------------------------
Eric Blake (10):
qemu-img: Flush stdout before before potential stderr messages
block: Finish deprecation of 'qemu-img convert -n -o'
sheepdog: Add trivial backing_fmt support
vmdk: Add trivial backing_fmt support
qcow: Tolerate backing_fmt=

I see you renumbered my test 293 to 301 for conflict resolution, but did not update the commit message to call out the updated number ;)

Since we may have to do a v3 because of the 'truncate -s' issue in 114, you could touch that up too.

Quoted 9 lines
  tests/qemu-iotests/301           | 88 ++++++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/301.out       | 59 +++++++++++++++++++++++++++
  tests/qemu-iotests/common.filter | 62 ++++++++++++----------------
  tests/qemu-iotests/group         |  1 +
  146 files changed, 906 insertions(+), 472 deletions(-)

create mode 100755 tests/qemu-iotests/301
create mode 100644 tests/qemu-iotests/301.out

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#4 lore raw

PATCH 1/1 fixup! qemu-img: Deprecate use of -b without -F

In reply to Kevin Wolf

Port to BSD truncate.

---
 tests/qemu-iotests/114 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/114 +1 -1
index 143683381334..d0609c499388 100755
@@ -45,7 +45,7 @@ _unsupported_imgopts data_file
 # Intentionally specify backing file without backing format; demonstrate
 # the difference in warning messages when backing file could be probed.
 # Note that only a non-raw probe result will affect the resulting image.
-truncate --size=64M "$TEST_IMG.orig"
+truncate -s $((64 * 1024 * 1024)) "$TEST_IMG.orig"
 _make_test_img -b "$TEST_IMG.orig" 64M
 
 TEST_IMG="$TEST_IMG.base" _make_test_img 64M
Signature · 1 line
2.27.0
#5 lore raw

PATCH 1/1 fixup! qemu-img: Flush stdout before before potential stderr messages

In reply to Kevin Wolf

Max noticed that I didn't run iotests on -luks...

---
 tests/qemu-iotests/282.out | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tests/qemu-iotests/282.out +3 -3
index 5d079dabce1d..67db7ab15aff 100644
@@ -1,11 +1,11 @@
 QA output created by 282
 == Create non-UTF8 secret ==
 == Throws an error because of invalid UTF-8 secret ==
-qemu-img: vol.img: Data from secret sec0 is not valid UTF-8
 Formatting 'vol.img', fmt=luks size=4194304 key-secret=sec0
+qemu-img: vol.img: Data from secret sec0 is not valid UTF-8
 == Image file should not exist after the error ==
 == Create a stub image file and run qemu-img again ==
-qemu-img: vol.img: Data from secret sec0 is not valid UTF-8
 Formatting 'vol.img', fmt=luks size=4194304 key-secret=sec0
+qemu-img: vol.img: Data from secret sec0 is not valid UTF-8
 == Pre-existing image file should also be deleted after the error ==
- *** done
+*** done
Signature · 1 line
2.27.0
#6 lore raw

PATCH 1/1 fixup! iotests: Specify explicit backing format where sensible

In reply to Kevin Wolf

Max spotted that I didn't test -qed...

---
 tests/qemu-iotests/024     | 2 +-
 tests/qemu-iotests/024.out | 1 -
 tests/qemu-iotests/043     | 5 +++--
 tests/qemu-iotests/043.out | 4 ----
 4 files changed, 4 insertions(+), 8 deletions(-)
tests/qemu-iotests/024 +1 -1
index 3c5d57310e94..12aceb2d41f6 100755
@@ -180,7 +180,7 @@ $QEMU_IMG rebase -f "$IMGFMT" -b "$BASE_NEW_OREL" -F $IMGFMT "$OVERLAY_WREL"
 popd >/dev/null
 
 # Verify the backing path is correct
-TEST_IMG=$OVERLAY _img_info | grep '^backing file'
+TEST_IMG=$OVERLAY _img_info | grep '^backing file:'
 
 echo
 
tests/qemu-iotests/024.out +0 -1
index f4fd77fd5b4d..973a5a371119 100644
@@ -158,7 +158,6 @@ wrote 65536/65536 bytes at offset 196608
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 backing file: t.IMGFMT.base_new (actual path: TEST_DIR/subdir/t.IMGFMT.base_new)
-backing file format: IMGFMT
 
 read 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/043 +3 -2
index e4022b7044de..3271737f69cc 100755
@@ -82,11 +82,12 @@ _make_test_img -b "$TEST_IMG.2.base" -F $IMGFMT $size
 
 echo
 echo "== finite chain of length 3 (human) =="
-_img_info --backing-chain
+# Exclude backing format, since qed differs from qcow2 on what gets stored
+_img_info --backing-chain | grep -v '^backing file format:'
 
 echo
 echo "== finite chain of length 3 (json) =="
-_img_info --backing-chain --output=json
+_img_info --backing-chain --output=json | grep -v 'backing-filename-format'
 
 # success, all done
 echo "*** done"
tests/qemu-iotests/043.out +0 -4
index 7251e26d4ff9..63ecb2181690 100644
@@ -25,14 +25,12 @@ file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
 cluster_size: 65536
 backing file: TEST_DIR/t.IMGFMT.2.base
-backing file format: IMGFMT
 
 image: TEST_DIR/t.IMGFMT.2.base
 file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
 cluster_size: 65536
 backing file: TEST_DIR/t.IMGFMT.1.base
-backing file format: IMGFMT
 
 image: TEST_DIR/t.IMGFMT.1.base
 file format: IMGFMT
@@ -42,7 +40,6 @@ cluster_size: 65536
 == finite chain of length 3 (json) ==
 [
     {
-        "backing-filename-format": "IMGFMT",
         "virtual-size": 134217728,
         "filename": "TEST_DIR/t.IMGFMT",
         "cluster-size": 65536,
@@ -52,7 +49,6 @@ cluster_size: 65536
         "dirty-flag": false
     },
     {
-        "backing-filename-format": "IMGFMT",
         "virtual-size": 134217728,
         "filename": "TEST_DIR/t.IMGFMT.2.base",
         "cluster-size": 65536,
Signature · 1 line
2.27.0
#7 lore raw

Re: [PATCH] fixup! iotests: Specify explicit backing format where sensible

Am 13.07.2020 um 13:40 hat Eric Blake geschrieben:

Quoted 3 lines

Max spotted that I didn't test -qed...

I guess I have a new name now... :-)

Kevin^WMax

#8 lore raw

Re: [PATCH] fixup! iotests: Specify explicit backing format where sensible

Quoted 6 lines · Kevin 'Max' Wolf

Am 13.07.2020 um 13:40 hat Eric Blake geschrieben:

Quoted 3 lines

Max spotted that I didn't test -qed...

I guess I have a new name now... :-)

Sorry, too many replies on the thread, and I messed up on which one I was reading!

Quoted 2 lines

Kevin^WMax

Brown Paper Bag^W^W^WEric

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org