Skip to content

preallocate filter

Vladimir Sementsov-Ogievskiy qemu-devel 7 versions – Applied

Status from this message: Thanks, applied to my block-next branch (for 6.0):

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v7 Latest21 patches 20 repliesApplied
  2. v615 patches 21 repliesSuperseded
  3. v510 patches 19 repliesSuperseded
Earlier versions · 4
VersionPostedPatchesRepliesStatus
v410 patches3 repliesSuperseded
v312 patches16 repliesSuperseded
v29 patches1 replySuperseded
v15 patches11 repliesSuperseded

Changes in v7 from the cover letter · 16 lines

01: add Alberto's r-b
07: don't remove sentence from the comment
08: - drop extra "s->file_end = end;" line
- improve check/set perm handlers
09: add Max's r-b
10: add Max's r-b
11: new
12: - skip if preallocate unsupported
- drop auto and quick groups
13: new
14: - improve 'average' field of result spec
- drop extra 'dim = ...' line
15-18: new
19: a lot of changes
20: new
21: add results dump to json

Patches in v7 Applied

#Subject and filesSizeRepliesApplied
1/21block: simplify comment to BDRV_REQ_SERIALISINGinclude/block/block.h+1 -100 replies
2/21block/io.c: drop assertion on double waiting for request serialisationblock/io.c+1 -100 replies
3/21block/io: split out bdrv_find_conflicting_requestblock/io.c+41 -300 replies
4/21block/io: bdrv_wait_serialising_requests_locked: drop extra bs argblock/io.c+5 -50 replies
5/21block: bdrv_mark_request_serialising: split non-waiting functionblock/file-posix.c block/io.c include/block/block_int.h+26 -140 replies
6/21block: introduce BDRV_REQ_NO_WAIT flagblock/io.c include/block/block.h+18 -20 replies
7/21block: bdrv_check_perm(): process children anywayblock.c+5 -21 reply
8/21block: introduce preallocate filterblock/meson.build block/preallocate.c docs/system/qemu-block-drivers.rst.inc +1 more+605 -12 replies
9/21qemu-io: add preallocate mode parameter for truncate commandqemu-io-cmds.c+32 -140 replies
10/21iotests: qemu_io_silent: support --image-optstests/qemu-iotests/iotests.py+6 -10 replies
11/21iotests.py: execute_setup_common(): add required_fmts argumenttests/qemu-iotests/iotests.py+8 -11 reply
12/21iotests: add 298 to test new preallocate filter drivertests/qemu-iotests/298 tests/qemu-iotests/298.out tests/qemu-iotests/group+192 -01 reply
13/21scripts/simplebench: fix grammar: s/successed/succeeded/scripts/simplebench/simplebench.py+6 -61 reply
14/21scripts/simplebench: support iopsscripts/simplebench/simplebench.py+28 -101 reply
15/21scripts/simplebench: use standard deviation for +- errorscripts/simplebench/simplebench.py+6 -51 reply
16/21simplebench: rename ascii() to results_to_text()scripts/simplebench/bench-example.py scripts/simplebench/bench_write_req.py scripts/simplebench/simplebench.py+7 -71 reply
17/21simplebench: move results_to_text() into separate filescripts/simplebench/bench-example.py scripts/simplebench/bench_write_req.py scripts/simplebench/results_to_text.py +1 more+52 -331 reply
18/21simplebench/results_to_text: improve view of the tablescripts/simplebench/results_to_text.py+12 -12 replies
19/21simplebench/results_to_text: add difference line to the tablescripts/simplebench/results_to_text.py+60 -72 replies
20/21simplebench/results_to_text: make executablescripts/simplebench/results_to_text.py+14 -01 reply
21/21scripts/simplebench: add bench_prealloc.pyscripts/simplebench/bench_prealloc.py+132 -03 replies

Reviews trailers per patch

All 21 patches have a Reviewed-by or Acked-by.

PatchStefan HajnocziMax ReitzAlberto GarciaPaolo Bonzini
1/21R-b R-b
2/21R-b
3/21R-b
4/21R-b
5/21R-b
6/21R-b
7/21R-b
8/21R-b
9/21R-b
10/21R-b
11/21R-b
12/21R-b
13/21R-b
14/21R-b
15/21R-b
16/21R-b
17/21R-b
18/21A-b
19/21R-b
20/21R-b
21/21R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v7 thread

Cover letter [PATCH v7 00/21] preallocate filter

Hi all!

Here is a filter, which does preallocation on write.

v7:
01: add Alberto's r-b
07: don't remove sentence from the comment
08: - drop extra "s->file_end = end;" line
- improve check/set perm handlers
09: add Max's r-b
10: add Max's r-b
11: new
12: - skip if preallocate unsupported
- drop auto and quick groups
13: new
14: - improve 'average' field of result spec
- drop extra 'dim = ...' line
15-18: new
19: a lot of changes
20: new
21: add results dump to json

In Virtuozzo we have to deal with some custom distributed storage solution, where allocation is very-very expensive operation. We have to workaround it in Qemu, so here is a new filter.

Still, the filter shows good results for me even for xfs and ext4.

Here are results, produced by new benchmark (last several patches):

All results are in iops (larger means better)

----------------------------------  -------------  ------------
                                    no-prealloc    prealloc
ssd-ext4, aligned sequential 16k    2e+04          2.7e+04
                                                     +36%
hdd-ext4, aligned sequential 16k    4.8e+03 ± 5%   3.1e+03 ± 4%
                                                     -36%
ssd-xfs, aligned sequential 16k     1.6e+04 ± 5%   2.5e+04 ± 3%
                                                     +59%
hdd-xfs, aligned sequential 16k     4.7e+03 ± 22%  3.2e+03 ± 3%
                                                     -31%
ssd-ext4, unaligned sequential 64k  2e+04          2.7e+04
                                                     +36%
hdd-ext4, unaligned sequential 64k  4.9e+03 ± 2%   3.2e+03 ± 8%
                                                     -34%
ssd-xfs, unaligned sequential 64k   1.5e+04        2.6e+04 ± 2%
                                                     +69%
hdd-xfs, unaligned sequential 64k   5.2e+03 ± 2%   3.3e+03 ± 5%
                                                     -35%

---------------------------------- ------------- ------------

Note: it's on Fedora 30, kernel 5.6.13-100.fc30.x86_64

The tests are actually qemu-img bench, run like:

./qemu-img create -f qcow2 $img 16G

aligned:
./qemu-img bench -c 10000 -d 64 -f qcow2 -s 16k -t none -n -w $img

unaligned
./qemu-img bench -c 10000 -d 64 -f qcow2 -o 1k -s 64k -t none -n -w $img

and for preallocation, you'll drop -f qcow2, add --image-opts, and instead of just $img use
driver=qcow2,file.driver=preallocate,file.file.driver=file,file.file.filename=$img

Or, use new benchmark like this:

  ./bench_prealloc.py ../../build/qemu-img \
      ssd-ext4:/path/to/mount/point \
      ssd-xfs:/path2 hdd-ext4:/path3 hdd-xfs:/path4

Vladimir Sementsov-Ogievskiy (21):
block: simplify comment to BDRV_REQ_SERIALISING block/io.c: drop assertion on double waiting for request serialisation block/io: split out bdrv_find_conflicting_request block/io: bdrv_wait_serialising_requests_locked: drop extra bs arg
block: bdrv_mark_request_serialising: split non-waiting function
block: introduce BDRV_REQ_NO_WAIT flag
block: bdrv_check_perm(): process children anyway
block: introduce preallocate filter
qemu-io: add preallocate mode parameter for truncate command
iotests: qemu_io_silent: support --image-opts
iotests.py: execute_setup_common(): add required_fmts argument
iotests: add 298 to test new preallocate filter driver scripts/simplebench: fix grammar: s/successed/succeeded/ scripts/simplebench: support iops
scripts/simplebench: use standard deviation for +- error
simplebench: rename ascii() to results_to_text()
simplebench: move results_to_text() into separate file simplebench/results_to_text: improve view of the table simplebench/results_to_text: add difference line to the table simplebench/results_to_text: make executable scripts/simplebench: add bench_prealloc.py

 docs/system/qemu-block-drivers.rst.inc |  26 ++
 qapi/block-core.json                   |  20 +-
 include/block/block.h                  |  20 +-
 include/block/block_int.h              |   3 +-
 block.c                                |   7 +-
 block/file-posix.c                     |   2 +-
 block/io.c                             | 130 +++---
 block/preallocate.c                    | 559 +++++++++++++++++++++++++
 qemu-io-cmds.c                         |  46 +-
 block/meson.build                      |   1 +
 scripts/simplebench/bench-example.py   |   3 +-
 scripts/simplebench/bench_prealloc.py  | 132 ++++++
 scripts/simplebench/bench_write_req.py |   3 +-
 scripts/simplebench/results_to_text.py | 126 ++++++
 scripts/simplebench/simplebench.py     |  66 ++-
 tests/qemu-iotests/298                 | 186 ++++++++
 tests/qemu-iotests/298.out             |   5 +
 tests/qemu-iotests/group               |   1 +
 tests/qemu-iotests/iotests.py          |  16 +-
 19 files changed, 1225 insertions(+), 127 deletions(-)
 create mode 100644 block/preallocate.c
 create mode 100755 scripts/simplebench/bench_prealloc.py
 create mode 100755 scripts/simplebench/results_to_text.py
 create mode 100644 tests/qemu-iotests/298
 create mode 100644 tests/qemu-iotests/298.out
Signature · 1 line
2.21.3