Skip to content

block: Deal with filters

Max Reitz qemu-devel 8 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v8 Latest43 patches 5 repliesIn review
  2. v747 patches 125 repliesSuperseded
  3. v642 patches 93 repliesSuperseded
Earlier versions · 5
VersionPostedPatchesRepliesStatus
v542 patches70 repliesSuperseded
v411 patches24 repliesSuperseded
v312 patches6 repliesSuperseded
v211 patches15 repliesSuperseded
v111 patches1 replySuperseded

Changes in v8 from the cover letter · 39 lines

- Patch 1: Let bdrv_primary_child() assert that there’s only a single primary child

- Patch 2: Assert that bdrv_do_skip_filters() will not return a filter node

- Patch 3: Drop bdrv_unallocated_blocks_are_zero() hunk (that function no longer exists)

- Old patch 10: Dropped, it isn’t really important to let mirror-top support compressed writes, and bdrv_supports_compressed_writes() doesn’t really work with this patch

- Old patch 11: Dropped, it isn’t really important to let backup-top support compressed writes

- Patch 10: Rebase conflict in init_dirty_bitmap_migration() (The modified block is now nested one level deeper)

- Patch 11: Rebase conflict in bdrv_co_block_status() (bdrv_unallocated_blocks_are_zero() is no longer used)

- Patch 12:
- Fix documentation on what the backing node after streaming is going to be
- bdrv_change_backing_file() rebase conflict

- Patch 13: Try to clarify what bdrv_find_overlay() returns in its doc comment

- Patch 15: Optimize the overlay_bs finding loop

- Old patch 25: Rolled into the next patch (now patch 23)

- Patch 23:
- Inline two of the three functions introduced in the old patch 25
- The other one (bdrv_sum_allocated_file_size()) stays, but is static now
(Turns out we don’t need to let block drivers use any of these functions for their BlockDriver.bdrv_get_allocated_File_size().)

- Old patch 27: Dropped, just let blkverify be handled like any other filter, because it doesn’t matter, really (and this allows us squashing the previous to patches)

- Patch 24: Remove _filter_actual_image_size from iotest 184

- Patch 27: Reference output change due to 184 no longer invoking _filter_actual_image_size

- Patch 29:
- Let the commit message explain why base_overlay is introduced
- Drop @replaces_node_name from qmp_drive_mirror(), because it isn’t needed

- Patch 31:
- Let blk_commit_all() only commit those nodes that have a backing file; and ignore all filters on top of them (instead of just implicit filters)
- In commit_start: %s/\<perms\>/base_perms/
- Under commit_start.ro_cleanup: Restore the original backing file only if necessary, so we do not run into an abort because the backing chain is frozen

- Patch 33: Let img_rebase() refer to the unfiltered_bs in three more places

- Patch 35:
- Documentation modifications:
- Move a chunk of it up where it belongs (from below @backing-file to the general area of @block-commit)
- Explain that you now need to issue block-job-complete whenever there’s a writer on @top
- Also, @backing-file is no longer allowed whenever there’s a writer on @top
- In the code, this should cause a different error message than when @top is in the active layer, though
- Comment note: If @top is a root node without any writers on it, we still need to do an active commit, because everything else would be an incompatible change

- Patch 36: Trivial rebase conflict in block_int.h (necessary because the old patch 25 was dropped)

- Patch 37: Don’t try to fetch ImageInfo.backing-filename-format from the backing file open at runtime. If the image header does not specify a format, just leave it empty.

- Patch 40: Use format string instead of ''.format()

- Patch 42: Add -F $IMGFMT to _make_test_img -b

Patches in v8 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/43block: Add child access functionsblock.c include/block/block_int.h+132 -30 replies
2/43block: Add chain helper functionsblock.c include/block/block_int.h+65 -00 replies
3/43block: bdrv_cow_child() for bdrv_has_zero_init()block.c+1 -10 replies
4/43block: bdrv_set_backing_hd() is about bs->backingblock.c+2 -20 replies
5/43block: Include filters when freezing backing chainblock.c+39 -210 replies
6/43block: Drop bdrv_is_encrypted()block.c block/qapi.c include/block/block.h+1 -100 replies
7/43block: Add bdrv_supports_compressed_writes()block.c include/block/block.h+24 -00 replies
8/43throttle: Support compressed writesblock/throttle.c+10 -00 replies
9/43copy-on-read: Support compressed writesblock/copy-on-read.c+11 -00 replies
10/43block: Use bdrv_filter_(bs|child) where obviousblock.c block/io.c migration/block-dirty-bitmap.c+26 -200 replies
11/43block: Use CAFs in block status functionsblock/io.c+10 -90 replies
12/43stream: Deal with filtersblock/stream.c blockdev.c qapi/block-core.json+59 -210 replies
13/43block: Use CAFs when working with backing chainsblock.c+30 -140 replies
14/43block: Use bdrv_cow_child() in bdrv_co_truncate()block/io.c+4 -30 replies
15/43block: Re-evaluate backing file handling in reopenblock.c+40 -90 replies
16/43block: Flush all children in generic codeblock/io.c+17 -60 replies
17/43vmdk: Drop vmdk_co_flush()block/vmdk.c+0 -160 replies
18/43block: Iterate over children in refresh_limitsblock/io.c+16 -160 replies
19/43block: Use CAFs in bdrv_refresh_filename()block.c+21 -80 replies
20/43block: Use CAF in bdrv_co_rw_vmstate()block/io.c+3 -20 replies
21/43block/snapshot: Fix fallbackblock/snapshot.c+83 -210 replies
22/43block: Use CAFs for debug breakpointsblock.c+3 -130 replies
23/43block: Improve get_allocated_file_size's defaultblock.c+39 -30 replies
24/43block/null: Implement bdrv_get_allocated_file_sizeblock/null.c tests/qemu-iotests/153.out tests/qemu-iotests/184 +1 more+13 -50 replies
25/43blockdev: Use CAF in external_snapshot_prepare()blockdev.c+6 -10 replies
26/43block: Report data child for query-blockstatsblock/qapi.c+29 -20 replies
27/43block: Use child access functions for QAPI queriesblock/qapi.c tests/qemu-iotests/184.out+27 -140 replies
28/43block-copy: Use CAF to find sync=top baseblock/block-copy.c+2 -20 replies
29/43mirror: Deal with filtersblock/mirror.c blockdev.c qapi/block-core.json+118 -382 replies
30/43backup: Deal with filtersblock/backup-top.c block/backup.c blockdev.c+21 -90 replies
31/43commit: Deal with filtersblock/block-backend.c block/commit.c block/monitor/block-hmp-cmds.c +1 more+79 -280 replies
32/43nbd: Use CAF when looking for dirty bitmapnbd/server.c+3 -30 replies
33/43qemu-img: Use child access functionsqemu-img.c tests/qemu-iotests/204.out+27 -170 replies
34/43block: Drop backing_bs()include/block/block_int.h+0 -50 replies
35/43blockdev: Fix active commit choiceblockdev.c qapi/block-core.json+50 -241 reply
36/43block: Inline bdrv_co_block_status_from_*()block/backup-top.c block/blkdebug.c block/blklogwrites.c +7 more+22 -680 replies
37/43block: Leave BDS.backing_{file,format} constantblock.c block/qapi.c include/block/block_int.h +4 more+59 -250 replies
38/43iotests: Test that qcow2's data-file is flushedtests/qemu-iotests/244 tests/qemu-iotests/244.out+56 -00 replies
39/43iotests: Let complete_and_wait() work with committests/qemu-iotests/iotests.py+7 -30 replies
40/43iotests: Add filter commit test casestests/qemu-iotests/040 tests/qemu-iotests/040.out+179 -20 replies
41/43iotests: Add filter mirror test casestests/qemu-iotests/041 tests/qemu-iotests/041.out+147 -30 replies
42/43iotests: Add test for commit in sub directorytests/qemu-iotests/020 tests/qemu-iotests/020.out+54 -00 replies
43/43iotests: Test committing to overridden backingtests/qemu-iotests/040 tests/qemu-iotests/040.out+63 -20 replies

Reviews trailers per patch

29 of 43 patches have a Reviewed-by or Acked-by.

PatchVladimir Sementsov-OgievskiyAndrey ShinkevichKevin Wolf
1/43R-b
2/43
3/43R-b R-b
4/43R-b R-b R-b
5/43R-b R-b
6/43R-b R-b
7/43R-b R-b
8/43R-b R-b R-b
9/43R-b R-b
10/43R-b R-b
11/43
12/43
13/43
14/43R-b R-b
15/43
16/43R-b R-b
17/43R-b
18/43R-b
19/43R-b
20/43R-b R-b R-b
21/43R-b R-b
22/43R-b R-b
23/43
24/43
25/43R-b R-b
26/43R-b R-b
27/43R-b R-b
28/43R-b R-b
29/43
30/43R-b
31/43
32/43R-b R-b
33/43
34/43R-b R-b R-b
35/43
36/43R-b R-b
37/43
38/43R-b R-b
39/43R-b R-b
40/43
41/43R-b R-b
42/43
43/43R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v8 thread

Cover letter [PATCH v8 00/43] block: Deal with filters

v6: https://lists.nongnu.org/archive/html/qemu-devel/2019-08/msg01715.html
v7: https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg01357.html

Branch: https://github.com/XanClic/qemu.git child-access-functions-v8
Branch: https://git.xanclic.moe/XanClic/qemu.git child-access-functions-v8

Hi,

In v8, there is not too much that has changed in respect to v7. I tried to address all of your comments and hope I got it right. I also hope I got the R-bs right.

Changes from v7:
- Patch 1: Let bdrv_primary_child() assert that there’s only a single primary child

- Patch 2: Assert that bdrv_do_skip_filters() will not return a filter node

- Patch 3: Drop bdrv_unallocated_blocks_are_zero() hunk (that function no longer exists)

- Old patch 10: Dropped, it isn’t really important to let mirror-top support compressed writes, and bdrv_supports_compressed_writes() doesn’t really work with this patch

- Old patch 11: Dropped, it isn’t really important to let backup-top support compressed writes

- Patch 10: Rebase conflict in init_dirty_bitmap_migration() (The modified block is now nested one level deeper)

- Patch 11: Rebase conflict in bdrv_co_block_status() (bdrv_unallocated_blocks_are_zero() is no longer used)

- Patch 12:
- Fix documentation on what the backing node after streaming is going to be
- bdrv_change_backing_file() rebase conflict

- Patch 13: Try to clarify what bdrv_find_overlay() returns in its doc comment

- Patch 15: Optimize the overlay_bs finding loop

- Old patch 25: Rolled into the next patch (now patch 23)

- Patch 23:
- Inline two of the three functions introduced in the old patch 25
- The other one (bdrv_sum_allocated_file_size()) stays, but is static now
(Turns out we don’t need to let block drivers use any of these functions for their BlockDriver.bdrv_get_allocated_File_size().)

- Old patch 27: Dropped, just let blkverify be handled like any other filter, because it doesn’t matter, really (and this allows us squashing the previous to patches)

- Patch 24: Remove _filter_actual_image_size from iotest 184

- Patch 27: Reference output change due to 184 no longer invoking _filter_actual_image_size

- Patch 29:
- Let the commit message explain why base_overlay is introduced
- Drop @replaces_node_name from qmp_drive_mirror(), because it isn’t needed

- Patch 31:
- Let blk_commit_all() only commit those nodes that have a backing file; and ignore all filters on top of them (instead of just implicit filters)
- In commit_start: %s/\<perms\>/base_perms/
- Under commit_start.ro_cleanup: Restore the original backing file only if necessary, so we do not run into an abort because the backing chain is frozen

- Patch 33: Let img_rebase() refer to the unfiltered_bs in three more places

- Patch 35:
- Documentation modifications:
- Move a chunk of it up where it belongs (from below @backing-file to the general area of @block-commit)
- Explain that you now need to issue block-job-complete whenever there’s a writer on @top
- Also, @backing-file is no longer allowed whenever there’s a writer on @top
- In the code, this should cause a different error message than when @top is in the active layer, though
- Comment note: If @top is a root node without any writers on it, we still need to do an active commit, because everything else would be an incompatible change

- Patch 36: Trivial rebase conflict in block_int.h (necessary because the old patch 25 was dropped)

- Patch 37: Don’t try to fetch ImageInfo.backing-filename-format from the backing file open at runtime. If the image header does not specify a format, just leave it empty.

- Patch 40: Use format string instead of ''.format()

- Patch 42: Add -F $IMGFMT to _make_test_img -b

git-backport-diff against v7:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/43:[0007] [FC] 'block: Add child access functions' 002/43:[0007] [FC] 'block: Add chain helper functions' 003/43:[0002] [FC] 'block: bdrv_cow_child() for bdrv_has_zero_init()' 004/43:[----] [--] 'block: bdrv_set_backing_hd() is about bs->backing' 005/43:[----] [--] 'block: Include filters when freezing backing chain' 006/43:[----] [--] 'block: Drop bdrv_is_encrypted()' 007/43:[----] [--] 'block: Add bdrv_supports_compressed_writes()' 008/43:[----] [--] 'throttle: Support compressed writes' 009/43:[----] [--] 'copy-on-read: Support compressed writes' 010/43:[0016] [FC] 'block: Use bdrv_filter_(bs|child) where obvious' 011/43:[0008] [FC] 'block: Use CAFs in block status functions' 012/43:[0017] [FC] 'stream: Deal with filters'
013/43:[0007] [FC] 'block: Use CAFs when working with backing chains' 014/43:[----] [--] 'block: Use bdrv_cow_child() in bdrv_co_truncate()' 015/43:[0009] [FC] 'block: Re-evaluate backing file handling in reopen' 016/43:[----] [--] 'block: Flush all children in generic code' 017/43:[----] [--] 'vmdk: Drop vmdk_co_flush()' 018/43:[----] [--] 'block: Iterate over children in refresh_limits' 019/43:[----] [--] 'block: Use CAFs in bdrv_refresh_filename()' 020/43:[----] [--] 'block: Use CAF in bdrv_co_rw_vmstate()' 021/43:[----] [--] 'block/snapshot: Fix fallback' 022/43:[----] [--] 'block: Use CAFs for debug breakpoints' 023/43:[0031] [FC] 'block: Improve get_allocated_file_size's default' 024/43:[0007] [FC] 'block/null: Implement bdrv_get_allocated_file_size' 025/43:[----] [--] 'blockdev: Use CAF in external_snapshot_prepare()' 026/43:[----] [--] 'block: Report data child for query-blockstats' 027/43:[0002] [FC] 'block: Use child access functions for QAPI queries' 028/43:[----] [--] 'block-copy: Use CAF to find sync=top base' 029/43:[0004] [FC] 'mirror: Deal with filters'
030/43:[----] [--] 'backup: Deal with filters'
031/43:[0020] [FC] 'commit: Deal with filters'
032/43:[----] [--] 'nbd: Use CAF when looking for dirty bitmap' 033/43:[0007] [FC] 'qemu-img: Use child access functions' 034/43:[----] [--] 'block: Drop backing_bs()'
035/43:[0052] [FC] 'blockdev: Fix active commit choice' 036/43:[0001] [FC] 'block: Inline bdrv_co_block_status_from_*()' 037/43:[0013] [FC] 'block: Leave BDS.backing_file constant' 038/43:[----] [--] 'iotests: Test that qcow2's data-file is flushed' 039/43:[----] [--] 'iotests: Let complete_and_wait() work with commit' 040/43:[0004] [FC] 'iotests: Add filter commit test cases' 041/43:[----] [--] 'iotests: Add filter mirror test cases' 042/43:[0008] [FC] 'iotests: Add test for commit in sub directory' 043/43:[----] [--] 'iotests: Test committing to overridden backing'

Max Reitz (43):
block: Add child access functions
block: Add chain helper functions
block: bdrv_cow_child() for bdrv_has_zero_init()
block: bdrv_set_backing_hd() is about bs->backing
block: Include filters when freezing backing chain
block: Drop bdrv_is_encrypted()
block: Add bdrv_supports_compressed_writes()
throttle: Support compressed writes
copy-on-read: Support compressed writes
block: Use bdrv_filter_(bs|child) where obvious
block: Use CAFs in block status functions
stream: Deal with filters
block: Use CAFs when working with backing chains
block: Use bdrv_cow_child() in bdrv_co_truncate()
block: Re-evaluate backing file handling in reopen
block: Flush all children in generic code
vmdk: Drop vmdk_co_flush()
block: Iterate over children in refresh_limits
block: Use CAFs in bdrv_refresh_filename()
block: Use CAF in bdrv_co_rw_vmstate() block/snapshot: Fix fallback
block: Use CAFs for debug breakpoints
block: Improve get_allocated_file_size's default block/null: Implement bdrv_get_allocated_file_size
blockdev: Use CAF in external_snapshot_prepare()
block: Report data child for query-blockstats
block: Use child access functions for QAPI queries
block-copy: Use CAF to find sync=top base
mirror: Deal with filters
backup: Deal with filters
commit: Deal with filters
nbd: Use CAF when looking for dirty bitmap
qemu-img: Use child access functions
block: Drop backing_bs()
blockdev: Fix active commit choice
block: Inline bdrv_co_block_status_from_*()
block: Leave BDS.backing_{file,format} constant
iotests: Test that qcow2's data-file is flushed
iotests: Let complete_and_wait() work with commit
iotests: Add filter commit test cases
iotests: Add filter mirror test cases
iotests: Add test for commit in sub directory
iotests: Test committing to overridden backing

 qapi/block-core.json           |  58 ++--
 include/block/block.h          |   2 +-
 include/block/block_int.h      |  95 ++++---
 block.c                        | 488 ++++++++++++++++++++++++++-------
 block/backup-top.c             |   4 +-
 block/backup.c                 |   9 +-
 block/blkdebug.c               |   7 +-
 block/blklogwrites.c           |   1 -
 block/block-backend.c          |   7 +-
 block/block-copy.c             |   4 +-
 block/commit.c                 |  95 +++++--
 block/copy-on-read.c           |  13 +-
 block/filter-compress.c        |   2 -
 block/io.c                     | 142 +++++-----
 block/mirror.c                 | 119 ++++++--
 block/monitor/block-hmp-cmds.c |   2 +-
 block/null.c                   |   7 +
 block/qapi.c                   |  74 +++--
 block/snapshot.c               | 104 +++++--
 block/stream.c                 |  63 +++--
 block/throttle.c               |  11 +-
 block/vmdk.c                   |  16 --
 blockdev.c                     | 101 +++++--
 migration/block-dirty-bitmap.c |   8 +-
 nbd/server.c                   |   6 +-
 qemu-img.c                     |  43 +--
 tests/qemu-iotests/020         |  44 +++
 tests/qemu-iotests/020.out     |  10 +
 tests/qemu-iotests/040         | 238 ++++++++++++++++
 tests/qemu-iotests/040.out     |   4 +-
 tests/qemu-iotests/041         | 146 +++++++++-
 tests/qemu-iotests/041.out     |   4 +-
 tests/qemu-iotests/153.out     |   2 +-
 tests/qemu-iotests/184         |   3 +-
 tests/qemu-iotests/184.out     |  14 +-
 tests/qemu-iotests/204.out     |   1 +
 tests/qemu-iotests/228         |   6 +-
 tests/qemu-iotests/228.out     |   6 +-
 tests/qemu-iotests/244         |  49 ++++
 tests/qemu-iotests/244.out     |   7 +
 tests/qemu-iotests/245         |   4 +-
 tests/qemu-iotests/273.out     |   4 +-
 tests/qemu-iotests/iotests.py  |  10 +-
 43 files changed, 1591 insertions(+), 442 deletions(-)
Signature · 1 line
2.26.2