Skip to contentSkip to conversation

qcow2, raw: add preallocation=full

PATCH Superseded Hu Tao 6 participants 39 messages Aug 29, 2014 – Sep 9, 2014 Block layer

v13 of 15 Superseded Latest is v15 →

lore
Share & tools
lore

Hu Tao

author
#0 lore raw

v13 cover qcow2, raw: add preallocation=full

This series adds preallocation=full to qcow2 and raw:

Option preallocation=full preallocates disk space for image by calling posix_fallocate() if it's available, otherwise by writing zeros to disk.

Max, Eric, Fam,

I removed your Reviewed-by line in all patches since they've been changed. You're welcome to review again!

changes to v12:

- remove dependence on minimal_blob_size() (patch 6)
- remove preallocation=falloc. (patch 4)
- preallocation=full tries posix_fallocate() first then writing zeros (patch 5)
- round up file size for all formats (patch 1)
- avoid converting file size for more formats (patch 2)

Hu Tao (5):
block: round up file size to nearest sector
block: don't convert file size to sector size
qapi: introduce PreallocMode and a new PreallocMode full.
raw-posix: Add full preallocation option
qcow2: Add full preallocation option

Peter Lieven (1):
rename parse_enum_option to qapi_enum_parse and make it public

 block/archipelago.c              |   3 +-
 block/cow.c                      |   3 +-
 block/gluster.c                  |   9 ++--
 block/iscsi.c                    |   4 +-
 block/nfs.c                      |   3 +-
 block/qcow.c                     |   7 +--
 block/qcow2.c                    |  79 +++++++++++++++++++++++++------
 block/qed.c                      |   3 +-
 block/raw-posix.c                | 100 +++++++++++++++++++++++++++++----------
 block/raw-win32.c                |   6 +--
 block/rbd.c                      |   3 +-
 block/sheepdog.c                 |   3 +-
 block/ssh.c                      |   3 +-
 block/vdi.c                      |   3 +-
 block/vhdx.c                     |   3 +-
 block/vmdk.c                     |   3 +-
 block/vpc.c                      |   3 +-
 blockdev.c                       |  30 +++---------
 include/qapi/util.h              |  17 +++++++
 qapi/Makefile.objs               |   1 +
 qapi/block-core.json             |  16 +++++++
 qapi/qapi-util.c                 |  34 +++++++++++++
 qemu-doc.texi                    |  15 ++++--
 qemu-img.texi                    |  15 ++++--
 tests/qemu-iotests/049.out       |   2 +-
 tests/qemu-iotests/082.out       |  54 ++++++++++-----------
 tests/qemu-iotests/104           |  57 ++++++++++++++++++++++
 tests/qemu-iotests/104.out       |  12 +++++
 tests/qemu-iotests/common.filter |  21 ++++++++
 tests/qemu-iotests/group         |   1 +
 30 files changed, 393 insertions(+), 120 deletions(-)
 create mode 100644 include/qapi/util.h
 create mode 100644 qapi/qapi-util.c
 create mode 100755 tests/qemu-iotests/104
 create mode 100644 tests/qemu-iotests/104.out
Signature · 1 line
1.9.3
author
#1 lore raw

PATCH v13 1/6 block: round up file size to nearest sector

---
 block/archipelago.c              |  3 ++-
 block/cow.c                      |  3 ++-
 block/gluster.c                  |  4 +--
 block/iscsi.c                    |  4 +--
 block/nfs.c                      |  3 ++-
 block/qcow.c                     |  3 ++-
 block/qcow2.c                    |  3 ++-
 block/qed.c                      |  3 ++-
 block/raw-posix.c                |  8 +++---
 block/raw-win32.c                |  4 +--
 block/rbd.c                      |  3 ++-
 block/sheepdog.c                 |  3 ++-
 block/ssh.c                      |  3 ++-
 block/vdi.c                      |  3 ++-
 block/vhdx.c                     |  3 ++-
 block/vmdk.c                     |  3 ++-
 block/vpc.c                      |  3 ++-
 tests/qemu-iotests/104           | 57 ++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/104.out       | 12 +++++++++
 tests/qemu-iotests/common.filter | 21 +++++++++++++++
 tests/qemu-iotests/group         |  1 +
 21 files changed, 127 insertions(+), 23 deletions(-)
 create mode 100755 tests/qemu-iotests/104
 create mode 100644 tests/qemu-iotests/104.out
block/archipelago.c +2 -1
index 34f72dc..a8d06aa 100644
@@ -707,7 +707,8 @@ static int qemu_archipelago_create(const char *filename,
 
     parse_filename_opts(filename, errp, &volname, &segment_name, &mport,
                         &vport);
-    total_size = qemu_opt_get_size_del(options, BLOCK_OPT_SIZE, 0);
+    total_size = ROUND_UP(qemu_opt_get_size_del(options, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
 
     if (segment_name == NULL) {
         segment_name = g_strdup("archipelago");
block/cow.c +2 -1
index 6ee4833..c3769fe 100644
@@ -335,7 +335,8 @@ static int cow_create(const char *filename, QemuOpts *opts, Error **errp)
     BlockDriverState *cow_bs = NULL;
 
     /* Read out options */
-    image_sectors = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / 512;
+    image_sectors = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                                 BDRV_SECTOR_SIZE);
     image_filename = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
 
     ret = bdrv_create_file(filename, opts, &local_err);
block/gluster.c +2 -2
index 1912cf9..65c7a58 100644
@@ -494,8 +494,8 @@ static int qemu_gluster_create(const char *filename,
         goto out;
     }
 
-    total_size =
-        qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / BDRV_SECTOR_SIZE;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
 
     tmp = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
     if (!tmp || !strcmp(tmp, "off")) {
block/iscsi.c +2 -2
index 3e19202..84bcae8 100644
@@ -1531,8 +1531,8 @@ static int iscsi_create(const char *filename, QemuOpts *opts, Error **errp)
     bs = bdrv_new("", &error_abort);
 
     /* Read out options */
-    total_size =
-        qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / BDRV_SECTOR_SIZE;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
     bs->opaque = g_new0(struct IscsiLun, 1);
     iscsilun = bs->opaque;
 
block/nfs.c +2 -1
index 93d87f3..d3ba09e 100644
@@ -414,7 +414,8 @@ static int nfs_file_create(const char *url, QemuOpts *opts, Error **errp)
     client->aio_context = qemu_get_aio_context();
 
     /* Read out options */
-    total_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
 
     ret = nfs_client_open(client, url, O_CREAT, errp);
     if (ret < 0) {
block/qcow.c +2 -1
index 67c237f..041af26 100644
@@ -725,7 +725,8 @@ static int qcow_create(const char *filename, QemuOpts *opts, Error **errp)
     BlockDriverState *qcow_bs;
 
     /* Read out options */
-    total_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / 512;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
     backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
     if (qemu_opt_get_bool_del(opts, BLOCK_OPT_ENCRYPT, false)) {
         flags |= BLOCK_FLAG_ENCRYPT;
block/qcow2.c +2 -1
index f9e045f..c8050e5 100644
@@ -1921,7 +1921,8 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     int ret;
 
     /* Read out options */
-    sectors = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / 512;
+    sectors = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                           BDRV_SECTOR_SIZE);
     backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
     backing_fmt = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FMT);
     if (qemu_opt_get_bool_del(opts, BLOCK_OPT_ENCRYPT, false)) {
block/qed.c +2 -1
index ba395af..f8d9e12 100644
@@ -648,7 +648,8 @@ static int bdrv_qed_create(const char *filename, QemuOpts *opts, Error **errp)
     char *backing_fmt = NULL;
     int ret;
 
-    image_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    image_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
     backing_fmt = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FMT);
     cluster_size = qemu_opt_get_size_del(opts,
block/raw-posix.c +4 -4
index d737f3a..9c22e3f 100644
@@ -1369,8 +1369,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
     /* Read out options */
-    total_size =
-        qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / BDRV_SECTOR_SIZE;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
 
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
@@ -1966,8 +1966,8 @@ static int hdev_create(const char *filename, QemuOpts *opts,
     (void)has_prefix;
 
     /* Read out options */
-    total_size =
-        qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / BDRV_SECTOR_SIZE;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
 
     fd = qemu_open(filename, O_WRONLY | O_BINARY);
     if (fd < 0) {
block/raw-win32.c +2 -2
index 902eab6..1e1880d 100644
@@ -511,8 +511,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
     /* Read out options */
-    total_size =
-        qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / 512;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
 
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
                    0644);
block/rbd.c +2 -1
index ea969e7..b7f7d5f 100644
@@ -314,7 +314,8 @@ static int qemu_rbd_create(const char *filename, QemuOpts *opts, Error **errp)
     }
 
     /* Read out options */
-    bytes = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    bytes = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                     BDRV_SECTOR_SIZE);
     objsize = qemu_opt_get_size_del(opts, BLOCK_OPT_CLUSTER_SIZE, 0);
     if (objsize) {
         if ((objsize - 1) & objsize) {    /* not a power of 2? */
block/sheepdog.c +2 -1
index 12cbd9d..8293d47 100644
@@ -1695,7 +1695,8 @@ static int sd_create(const char *filename, QemuOpts *opts,
         goto out;
     }
 
-    s->inode.vdi_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    s->inode.vdi_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                                 BDRV_SECTOR_SIZE);
     backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
     buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
     if (!buf || !strcmp(buf, "off")) {
block/ssh.c +2 -1
index cd2fd75..cf43bc0 100644
@@ -700,7 +700,8 @@ static int ssh_create(const char *filename, QemuOpts *opts, Error **errp)
     ssh_state_init(&s);
 
     /* Get desired file size. */
-    total_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     DPRINTF("total_size=%" PRIi64, total_size);
 
     uri_options = qdict_new();
block/vdi.c +2 -1
index 4b10aac..cfa08b0 100644
@@ -700,7 +700,8 @@ static int vdi_create(const char *filename, QemuOpts *opts, Error **errp)
     logout("\n");
 
     /* Read out options. */
-    bytes = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    bytes = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                     BDRV_SECTOR_SIZE);
 #if defined(CONFIG_VDI_BLOCK_SIZE)
     /* TODO: Additional checks (SECTOR_SIZE * 2^n, ...). */
     block_size = qemu_opt_get_size_del(opts,
block/vhdx.c +2 -1
index 87c99fc..796b7bd 100644
@@ -1766,7 +1766,8 @@ static int vhdx_create(const char *filename, QemuOpts *opts, Error **errp)
     VHDXImageType image_type;
     Error *local_err = NULL;
 
-    image_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    image_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     log_size = qemu_opt_get_size_del(opts, VHDX_BLOCK_OPT_LOG_SIZE, 0);
     block_size = qemu_opt_get_size_del(opts, VHDX_BLOCK_OPT_BLOCK_SIZE, 0);
     type = qemu_opt_get_del(opts, BLOCK_OPT_SUBFMT);
block/vmdk.c +2 -1
index 07cb62c..2e126c5 100644
@@ -1805,7 +1805,8 @@ static int vmdk_create(const char *filename, QemuOpts *opts, Error **errp)
         goto exit;
     }
     /* Read out options */
-    total_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     adapter_type = qemu_opt_get_del(opts, BLOCK_OPT_ADAPTER_TYPE);
     backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
     if (qemu_opt_get_bool_del(opts, BLOCK_OPT_COMPAT6, false)) {
block/vpc.c +2 -1
index 055efc4..5d8fd8e 100644
@@ -757,7 +757,8 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp)
     BlockDriverState *bs = NULL;
 
     /* Read out options */
-    total_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     disk_type_param = qemu_opt_get_del(opts, BLOCK_OPT_SUBFMT);
     if (disk_type_param) {
         if (!strcmp(disk_type_param, "dynamic")) {
tests/qemu-iotests/104 new file +57 -0

57 lines not on this page · raw message

tests/qemu-iotests/104.out new file +12 -0
new file mode 100644
index 0000000..de27852
@@ -0,0 +1,12 @@
+QA output created by 104
+=== Check qemu-img info output ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1024 
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 1.0K (1024 bytes)
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1234 
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 1.5K (1536 bytes)
+***done
tests/qemu-iotests/common.filter +21 -0
index 51192c8..f69cb6b 100644
@@ -192,5 +192,26 @@ _filter_img_create()
         -e "s/archipelago:a/TEST_DIR\//g"
 }
 
+_filter_img_info()
+{
+    sed -e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \
+        -e "s#$TEST_DIR#TEST_DIR#g" \
+        -e "s#$IMGFMT#IMGFMT#g" \
+        -e "/encrypted: yes/d" \
+        -e "/cluster_size: [0-9]\\+/d" \
+        -e "/table_size: [0-9]\\+/d" \
+        -e "/compat: '[^']*'/d" \
+        -e "/compat6: \\(on\\|off\\)/d" \
+        -e "/static: \\(on\\|off\\)/d" \
+        -e "/zeroed_grain: \\(on\\|off\\)/d" \
+        -e "/subformat: '[^']*'/d" \
+        -e "/adapter_type: '[^']*'/d" \
+        -e "/lazy_refcounts: \\(on\\|off\\)/d" \
+        -e "/block_size: [0-9]\\+/d" \
+        -e "/block_state_zero: \\(on\\|off\\)/d" \
+        -e "/log_size: [0-9]\\+/d" \
+        -e "s/archipelago:a/TEST_DIR\//g"
+}
+
 # make sure this script returns success
 /bin/true
tests/qemu-iotests/group +1 -0
index 2803d68..e6c23b4 100644
@@ -103,3 +103,4 @@
 099 rw auto quick
 101 rw auto quick
 103 rw auto quick
+104 rw auto
Signature · 1 line
1.9.3
Reviewed-by on v13 1/6
#2 lore raw

Re: [Qemu-devel] [PATCH v13 1/6] block: round up file size to nearest sector

Quoted 18 lines · Hu Tao
---
 block/archipelago.c              |  3 ++-
 block/cow.c                      |  3 ++-
 block/gluster.c                  |  4 +--
 block/iscsi.c                    |  4 +--
 block/nfs.c                      |  3 ++-
 block/qcow.c                     |  3 ++-
 block/qcow2.c                    |  3 ++-
 block/qed.c                      |  3 ++-
 block/raw-posix.c                |  8 +++---
 block/raw-win32.c                |  4 +--
 block/rbd.c                      |  3 ++-
 block/sheepdog.c                 |  3 ++-
 block/ssh.c                      |  3 ++-
 block/vdi.c                      |  3 ++-
 block/vhdx.c                     |  3 ++-
 block/vmdk.c                     |  3 ++-
Quoted 8 lines
 block/vpc.c                      |  3 ++-
 tests/qemu-iotests/104           | 57 ++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/104.out       | 12 +++++++++
 tests/qemu-iotests/common.filter | 21 +++++++++++++++
 tests/qemu-iotests/group         |  1 +
 21 files changed, 127 insertions(+), 23 deletions(-)
 create mode 100755 tests/qemu-iotests/104
 create mode 100644 tests/qemu-iotests/104.out

Code looks correct. However, I have a possible design concern:

block/qcow2.c static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) · line 1925

Quoted 8 lines

+++ b/block/qcow2.c

block/qcow2.c +2 -1
@@ -1921,7 +1921,8 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     int ret;
 
     /* Read out options */
-    sectors = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / 512;
+    sectors = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                           BDRV_SECTOR_SIZE);

At least the qcow2 file format describes virtual file size in bytes (offsets 24-31 in the header). It's nice that we are guaranteeing at least the size requested by the user, but by doing the division, we are unable to write the user's actual requested size into the file.

I'm okay if all files _we_ create are aligned in size. But do we have any lurking bugs if a qcow2 file created by some other means has a non-aligned size? The recent work on the image fuzzer should be able to easily create such images. Meanwhile, I don't know of any physical hardware that has an unaligned size. Should we tighten the qcow2 spec to forbid a size that is not a multiple of the sector size, and teach qemu to forcefully reject such images as invalid? Does the qcow2 format need a way to flag whether an image has 512 vs. 4096-byte sectors (that is, what happens with an image that is 512-aligned but not 4096-aligned when used in an emulation that only exposes 4096-byte alignment to the guest)?

On the other hand, if we _do_ have code that gracefully handles unaligned size from externally-created sources, would it be worth allowing qemu-img to also create unaligned images directly, instead of having to resort to the image fuzzer to create such images, all so that it is easier to test our code and ensure it doesn't bit-rot? It might be lower-maintenance to just require that such images be rejected as broken.

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1374

Quoted 10 lines
block/raw-posix.c +3 -2
+++ b/block/raw-posix.c
@@ -1369,8 +1369,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
     /* Read out options */
-    total_size =
-        qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / BDRV_SECTOR_SIZE;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);

Again, I'm okay with the idea of always treating guest images as aligned multiples of sectors. But it's very easy to create an unaligned raw file, and pass that to qemu. So even though we don't create such files via qemu-img, it's still worth ensuring that the code behaves correctly when such an image is used directly or as a backing file (as a backing file of a larger qcow2 image, bytes beyond the actual size must read as padded to 0; as a direct file, I'm not sure that the guest OS can do a partial sector read, so I'd lean more towards an I/O error when trying to read a full sector when only a partial sector is available the same as when trying to read a sector completely beyond the disk bounds, but I could also live with a successful read with the tail being padded to 0; similarly, we'd have to ensure that writes either error out, or guarantee that the tail is ignored and only the valid portion of the sector is written).

Meanwhile, although we own the qcow2 spec, and can therefore choose to reject unaligned size in qcow2 files as invalid format, we are not in charge of the spec for several other file formats, but probably ought to handle those the same way as other vendors. For example, is it possible to create a VMDK file with unaligned size, and if so, how does it behave?

tests/qemu-iotests/common.filter _filter_img_create() · line 197

Quoted 4 lines
tests/qemu-iotests/common.filter +3 -0
+_filter_img_info()
+{
+    sed -e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \

This works, so I'm not asking for a change; but I prefer | over # when writing a sed expression that includes a filename expansion, as in:

sed -e "s|$IMGPROTO:$TEST_DIR|TEST_DIR|g"

Why? Because it's easy (even if uncommon) to create file names with # embedded in them (if $TEST_DIR is absolute, I just have to have /path/to/my#odd/directory with no shell quoting required as the place where I unpack and configure qemu), but file names with | can only be created with shell quoting, so they are less common. (The complaint gets more relevant for people doing s,,, with file names, because such files are more common [anyone remember CVS?])

At any rate, if there are still design issues to resolve (such as tightening the qcow2 spec), they can be separate patches. So I'm okay with:

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#3 lore raw

Re: [Qemu-devel] [PATCH v13 1/6] block: round up file size to nearest sector

Am 29.08.2014 um 14:50 hat Eric Blake geschrieben:

Quoted 5 lines · Hu Tao
Quoted 4 lines

+++ b/block/raw-posix.c

(quoted diff) +0 -0
@@ -1369,8 +1369,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
Quoted 12 lines

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1374

Quoted 6 lines
block/raw-posix.c +2 -2
     /* Read out options */
-    total_size =
-        qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0) / BDRV_SECTOR_SIZE;
+    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                              BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);

Again, I'm okay with the idea of always treating guest images as aligned multiples of sectors. But it's very easy to create an unaligned raw file, and pass that to qemu. So even though we don't create such files via qemu-img, it's still worth ensuring that the code behaves correctly when such an image is used directly or as a backing file

The whole qemu block layer doesn't work well with unaligned images (and the required alignment is always 512 because someone thought that using 512-byte units was a good interface). I want to fix that by converting everything to byte granularity, but it's not the top priority.

Kevin

Reviewed-by on v13 1/6
#4 lore raw

Re: [Qemu-devel] [PATCH v13 1/6] block: round up file size to nearest sector

In reply to Hu Tao
Quoted 24 lines · Hu Tao
---
  block/archipelago.c              |  3 ++-
  block/cow.c                      |  3 ++-
  block/gluster.c                  |  4 +--
  block/iscsi.c                    |  4 +--
  block/nfs.c                      |  3 ++-
  block/qcow.c                     |  3 ++-
  block/qcow2.c                    |  3 ++-
  block/qed.c                      |  3 ++-
  block/raw-posix.c                |  8 +++---
  block/raw-win32.c                |  4 +--
  block/rbd.c                      |  3 ++-
  block/sheepdog.c                 |  3 ++-
  block/ssh.c                      |  3 ++-
  block/vdi.c                      |  3 ++-
  block/vhdx.c                     |  3 ++-
  block/vmdk.c                     |  3 ++-
  block/vpc.c                      |  3 ++-
  tests/qemu-iotests/104           | 57 ++++++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/104.out       | 12 +++++++++
  tests/qemu-iotests/common.filter | 21 +++++++++++++++
  tests/qemu-iotests/group         |  1 +
  21 files changed, 127 insertions(+), 23 deletions(-)
Quoted 2 lines

create mode 100755 tests/qemu-iotests/104
create mode 100644 tests/qemu-iotests/104.out

Reviewed-by on v13 1/6
#5 lore raw

Re: [Qemu-devel] [PATCH v13 1/6] block: round up file size to nearest sector

In reply to Hu Tao

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

tests/qemu-iotests/104 line 3

Quoted 11 lines
tests/qemu-iotests/104 new file +3 -0
new file mode 100755
index 0000000..0c1d4fb
@@ -0,0 +1,57 @@
+#!/bin/bash
+#
+# Test qcow2 creation with aligned and unaligned sizes

This test case is not only for qcow2.

With this fixed, you can add:

author
#6 lore raw

PATCH v13 2/6 block: don't convert file size to sector size

In reply to Hu Tao

and avoid converting it back later.

---
 block/gluster.c   |  9 ++++-----
 block/qcow.c      |  8 ++++----
 block/qcow2.c     | 10 +++++-----
 block/raw-posix.c |  6 +++---
 block/raw-win32.c |  6 +++---
 5 files changed, 19 insertions(+), 20 deletions(-)
block/gluster.c +4 -5
index 65c7a58..1eb3a8c 100644
@@ -494,8 +494,8 @@ static int qemu_gluster_create(const char *filename,
         goto out;
     }
 
-    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                              BDRV_SECTOR_SIZE);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
 
     tmp = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
     if (!tmp || !strcmp(tmp, "off")) {
@@ -516,9 +516,8 @@ static int qemu_gluster_create(const char *filename,
     if (!fd) {
         ret = -errno;
     } else {
-        if (!glfs_ftruncate(fd, total_size * BDRV_SECTOR_SIZE)) {
-            if (prealloc && qemu_gluster_zerofill(fd, 0,
-                    total_size * BDRV_SECTOR_SIZE)) {
+        if (!glfs_ftruncate(fd, total_size)) {
+            if (prealloc && qemu_gluster_zerofill(fd, 0, total_size)) {
                 ret = -errno;
             }
         } else {
block/qcow.c +4 -4
index 041af26..a87bd69 100644
@@ -725,8 +725,8 @@ static int qcow_create(const char *filename, QemuOpts *opts, Error **errp)
     BlockDriverState *qcow_bs;
 
     /* Read out options */
-    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                              BDRV_SECTOR_SIZE);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
     if (qemu_opt_get_bool_del(opts, BLOCK_OPT_ENCRYPT, false)) {
         flags |= BLOCK_FLAG_ENCRYPT;
@@ -754,7 +754,7 @@ static int qcow_create(const char *filename, QemuOpts *opts, Error **errp)
     memset(&header, 0, sizeof(header));
     header.magic = cpu_to_be32(QCOW_MAGIC);
     header.version = cpu_to_be32(QCOW_VERSION);
-    header.size = cpu_to_be64(total_size * 512);
+    header.size = cpu_to_be64(total_size);
     header_size = sizeof(header);
     backing_filename_len = 0;
     if (backing_file) {
@@ -776,7 +776,7 @@ static int qcow_create(const char *filename, QemuOpts *opts, Error **errp)
     }
     header_size = (header_size + 7) & ~7;
     shift = header.cluster_bits + header.l2_bits;
-    l1_size = ((total_size * 512) + (1LL << shift) - 1) >> shift;
+    l1_size = (total_size + (1LL << shift) - 1) >> shift;
 
     header.l1_table_offset = cpu_to_be64(header_size);
     if (flags & BLOCK_FLAG_ENCRYPT) {
block/qcow2.c +5 -5
index c8050e5..cf27c3f 100644
@@ -1859,7 +1859,7 @@ static int qcow2_create2(const char *filename, int64_t total_size,
     }
 
     /* Okay, now that we have a valid image, let's give it the right size */
-    ret = bdrv_truncate(bs, total_size * BDRV_SECTOR_SIZE);
+    ret = bdrv_truncate(bs, total_size);
     if (ret < 0) {
         error_setg_errno(errp, -ret, "Could not resize image");
         goto out;
@@ -1912,7 +1912,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     char *backing_file = NULL;
     char *backing_fmt = NULL;
     char *buf = NULL;
-    uint64_t sectors = 0;
+    uint64_t size = 0;
     int flags = 0;
     size_t cluster_size = DEFAULT_CLUSTER_SIZE;
     int prealloc = 0;
@@ -1921,8 +1921,8 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     int ret;
 
     /* Read out options */
-    sectors = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                           BDRV_SECTOR_SIZE);
+    size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                    BDRV_SECTOR_SIZE);
     backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE);
     backing_fmt = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FMT);
     if (qemu_opt_get_bool_del(opts, BLOCK_OPT_ENCRYPT, false)) {
@@ -1972,7 +1972,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         goto finish;
     }
 
-    ret = qcow2_create2(filename, sectors, backing_file, backing_fmt, flags,
+    ret = qcow2_create2(filename, size, backing_file, backing_fmt, flags,
                         cluster_size, prealloc, opts, version, &local_err);
     if (local_err) {
         error_propagate(errp, local_err);
block/raw-posix.c +3 -3
index 9c22e3f..abe0759 100644
@@ -1369,8 +1369,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
     /* Read out options */
-    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                              BDRV_SECTOR_SIZE);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
 
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
@@ -1394,7 +1394,7 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
 #endif
         }
 
-        if (ftruncate(fd, total_size * BDRV_SECTOR_SIZE) != 0) {
+        if (ftruncate(fd, total_size) != 0) {
             result = -errno;
             error_setg_errno(errp, -result, "Could not resize file");
         }
block/raw-win32.c +3 -3
index 1e1880d..9bf8225 100644
@@ -511,8 +511,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
     /* Read out options */
-    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                              BDRV_SECTOR_SIZE);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
 
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
                    0644);
@@ -521,7 +521,7 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
         return -EIO;
     }
     set_sparse(fd);
-    ftruncate(fd, total_size * 512);
+    ftruncate(fd, total_size);
     qemu_close(fd);
     return 0;
 }
Signature · 1 line
1.9.3
Reviewed-by on v13 2/6
#7 lore raw

Re: [Qemu-devel] [PATCH v13 2/6] block: don't convert file size to sector size

Quoted 10 lines · Hu Tao

and avoid converting it back later.

---
  block/gluster.c   |  9 ++++-----
  block/qcow.c      |  8 ++++----
  block/qcow2.c     | 10 +++++-----
  block/raw-posix.c |  6 +++---
  block/raw-win32.c |  6 +++---
  5 files changed, 19 insertions(+), 20 deletions(-)
#8 lore raw

Re: [Qemu-devel] [PATCH v13 2/6] block: don't convert file size to sector size

In reply to Hu Tao

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

Quoted 20 lines

and avoid converting it back later.

block/raw-posix.c +2 -2
index 9c22e3f..abe0759 100644
@@ -1369,8 +1369,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
     /* Read out options */
-    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                              BDRV_SECTOR_SIZE);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
 
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1400

Quoted 8 lines
block/raw-posix.c +1 -1
@@ -1394,7 +1394,7 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
 #endif
         }
 
-        if (ftruncate(fd, total_size * BDRV_SECTOR_SIZE) != 0) {
+        if (ftruncate(fd, total_size) != 0) {
             result = -errno;
             error_setg_errno(errp, -result, "Could not resize file");
         }

You forgot changing hdev_create() in raw-posix. Doesn't make the patch less correct, but you may want to add it for v14.

Kevin

author
#9 lore raw

Re: [Qemu-devel] [PATCH v13 2/6] block: don't convert file size to sector size

Quoted 24 lines · Kevin Wolf

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

Quoted 23 lines

and avoid converting it back later.

block/raw-posix.c +2 -2
index 9c22e3f..abe0759 100644
@@ -1369,8 +1369,8 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     strstart(filename, "file:", &filename);
 
     /* Read out options */
-    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                              BDRV_SECTOR_SIZE);
+    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+                          BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
 
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
@@ -1394,7 +1394,7 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
 #endif
         }
 
Quoted 8 lines

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1400

Quoted 5 lines
block/raw-posix.c +1 -1
-        if (ftruncate(fd, total_size * BDRV_SECTOR_SIZE) != 0) {
+        if (ftruncate(fd, total_size) != 0) {
             result = -errno;
             error_setg_errno(errp, -result, "Could not resize file");
         }

You forgot changing hdev_create() in raw-posix. Doesn't make the patch less correct, but you may want to add it for v14.

Thanks, changed it too.

Regards,
Hu

Quoted 2 lines

Kevin

author
#10 lore raw

PATCH v13 3/6 rename parse_enum_option to qapi_enum_parse and make it public

In reply to Hu Tao

From: Peter Lieven <[email protected]>

relaxing the license to LGPLv2+ is intentional.

---
 blockdev.c          | 30 ++++++------------------------
 include/qapi/util.h | 17 +++++++++++++++++
 qapi/Makefile.objs  |  1 +
 qapi/qapi-util.c    | 34 ++++++++++++++++++++++++++++++++++
 4 files changed, 58 insertions(+), 24 deletions(-)
 create mode 100644 include/qapi/util.h
 create mode 100644 qapi/qapi-util.c
blockdev.c +6 -24
index 6a204c6..b2aa7a2 100644
@@ -39,6 +39,7 @@
 #include "qapi/qmp/types.h"
 #include "qapi-visit.h"
 #include "qapi/qmp-output-visitor.h"
+#include "qapi/util.h"
 #include "sysemu/sysemu.h"
 #include "block/block_int.h"
 #include "qmp-commands.h"
@@ -274,25 +275,6 @@ static int parse_block_error_action(const char *buf, bool is_read, Error **errp)
     }
 }
 
-static inline int parse_enum_option(const char *lookup[], const char *buf,
-                                    int max, int def, Error **errp)
-{
-    int i;
-
-    if (!buf) {
-        return def;
-    }
-
-    for (i = 0; i < max; i++) {
-        if (!strcmp(buf, lookup[i])) {
-            return i;
-        }
-    }
-
-    error_setg(errp, "invalid parameter value: %s", buf);
-    return def;
-}
-
 static bool check_throttle_config(ThrottleConfig *cfg, Error **errp)
 {
     if (throttle_conflicting(cfg)) {
@@ -456,11 +438,11 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts,
     }
 
     detect_zeroes =
-        parse_enum_option(BlockdevDetectZeroesOptions_lookup,
-                          qemu_opt_get(opts, "detect-zeroes"),
-                          BLOCKDEV_DETECT_ZEROES_OPTIONS_MAX,
-                          BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF,
-                          &error);
+        qapi_enum_parse(BlockdevDetectZeroesOptions_lookup,
+                        qemu_opt_get(opts, "detect-zeroes"),
+                        BLOCKDEV_DETECT_ZEROES_OPTIONS_MAX,
+                        BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF,
+                        &error);
     if (error) {
         error_propagate(errp, error);
         goto early_err;
include/qapi/util.h new file +17 -0
new file mode 100644
index 0000000..de9238b
@@ -0,0 +1,17 @@
+/*
+ * QAPI util functions
+ *
+ * Copyright Fujitsu, Inc. 2014
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef QAPI_UTIL_H
+#define QAPI_UTIL_H
+
+int qapi_enum_parse(const char *lookup[], const char *buf,
+                    int max, int def, Error **errp);
+
+#endif
qapi/Makefile.objs +1 -0
index d14b769..ffd88a6 100644
@@ -4,3 +4,4 @@ util-obj-y += string-input-visitor.o string-output-visitor.o
 
 util-obj-y += opts-visitor.o
 util-obj-y += qmp-event.o
+util-obj-y += qapi-util.o
qapi/qapi-util.c new file +34 -0
new file mode 100644
index 0000000..982d4e8
@@ -0,0 +1,34 @@
+/*
+ * QAPI util functions
+ *
+ * Authors:
+ *  Hu Tao       <[email protected]>
+ *  Peter Lieven <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "qapi/error.h"
+#include "qapi/util.h"
+
+int qapi_enum_parse(const char *lookup[], const char *buf,
+                    int max, int def, Error **errp)
+{
+    int i;
+
+    if (!buf) {
+        return def;
+    }
+
+    for (i = 0; i < max; i++) {
+        if (!strcmp(buf, lookup[i])) {
+            return i;
+        }
+    }
+
+    error_setg(errp, "invalid parameter value: %s", buf);
+    return def;
+}
Signature · 1 line
1.9.3
Reviewed-by on v13 3/6
#11 lore raw

Re: [Qemu-devel] [PATCH v13 3/6] rename parse_enum_option to qapi_enum_parse and make it public

Quoted 9 lines · Hu Tao

From: Peter Lieven <[email protected]>

relaxing the license to LGPLv2+ is intentional.

Quoted 8 lines
---
  blockdev.c          | 30 ++++++------------------------
  include/qapi/util.h | 17 +++++++++++++++++
  qapi/Makefile.objs  |  1 +
  qapi/qapi-util.c    | 34 ++++++++++++++++++++++++++++++++++
  4 files changed, 58 insertions(+), 24 deletions(-)

create mode 100644 include/qapi/util.h create mode 100644 qapi/qapi-util.c

Seems pretty much unchanged (except for the author note), so:

author
#12 lore raw

Re: [Qemu-devel] [PATCH v13 3/6] rename parse_enum_option to qapi_enum_parse and make it public

Quoted 19 lines · Max Reitz
Quoted 9 lines · Hu Tao

From: Peter Lieven <[email protected]>

relaxing the license to LGPLv2+ is intentional.

Quoted 8 lines
---
 blockdev.c          | 30 ++++++------------------------
 include/qapi/util.h | 17 +++++++++++++++++
 qapi/Makefile.objs  |  1 +
 qapi/qapi-util.c    | 34 ++++++++++++++++++++++++++++++++++
 4 files changed, 58 insertions(+), 24 deletions(-)
 create mode 100644 include/qapi/util.h
 create mode 100644 qapi/qapi-util.c
Quoted 1 line

Seems pretty much unchanged (except for the author note), so:

Yes. There was a license discussion, so I just take the latest version by Peter.

Regards,
Hu

Quoted 2 lines
Reviewed-by on v13 3/6
#13 lore raw

Re: [Qemu-devel] [PATCH v13 3/6] rename parse_enum_option to qapi_enum_parse and make it public

In reply to Hu Tao

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

Quoted 9 lines

From: Peter Lieven <[email protected]>

relaxing the license to LGPLv2+ is intentional.

author
#14 lore raw

PATCH v13 4/6 qapi: introduce PreallocMode and a new PreallocMode full.

In reply to Hu Tao

This patch prepares for the subsequent patches.

---
 block/qcow2.c              | 23 +++++++++++++++--------
 qapi/block-core.json       | 16 ++++++++++++++++
 tests/qemu-iotests/049.out |  2 +-
 3 files changed, 32 insertions(+), 9 deletions(-)
block/qcow2.c +15 -8
index cf27c3f..95fb240 100644
@@ -30,6 +30,7 @@
 #include "qemu/error-report.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
+#include "qapi/util.h"
 #include "trace.h"
 #include "qemu/option_int.h"
 
@@ -1738,7 +1739,7 @@ static int preallocate(BlockDriverState *bs)
 
 static int qcow2_create2(const char *filename, int64_t total_size,
                          const char *backing_file, const char *backing_format,
-                         int flags, size_t cluster_size, int prealloc,
+                         int flags, size_t cluster_size, PreallocMode prealloc,
                          QemuOpts *opts, int version,
                          Error **errp)
 {
@@ -1915,7 +1916,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     uint64_t size = 0;
     int flags = 0;
     size_t cluster_size = DEFAULT_CLUSTER_SIZE;
-    int prealloc = 0;
+    PreallocMode prealloc = PREALLOC_MODE_OFF;
     int version = 3;
     Error *local_err = NULL;
     int ret;
@@ -1931,12 +1932,11 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     cluster_size = qemu_opt_get_size_del(opts, BLOCK_OPT_CLUSTER_SIZE,
                                          DEFAULT_CLUSTER_SIZE);
     buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
-    if (!buf || !strcmp(buf, "off")) {
-        prealloc = 0;
-    } else if (!strcmp(buf, "metadata")) {
-        prealloc = 1;
-    } else {
-        error_setg(errp, "Invalid preallocation mode: '%s'", buf);
+    prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
+                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
         ret = -EINVAL;
         goto finish;
     }
@@ -1958,6 +1958,13 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         flags |= BLOCK_FLAG_LAZY_REFCOUNTS;
     }
 
+    if (prealloc && prealloc != PREALLOC_MODE_METADATA) {
+        ret = -1;
+        error_setg(errp, "Unsupported preallocate mode: %s",
+                   PreallocMode_lookup[prealloc]);
+        goto finish;
+    }
+
     if (backing_file && prealloc) {
         error_setg(errp, "Backing file and preallocation cannot be used at "
                    "the same time");
qapi/block-core.json +16 -0
index fb74c56..543b00b 100644
@@ -1679,3 +1679,19 @@
             'len'   : 'int',
             'offset': 'int',
             'speed' : 'int' } }
+
+# @PreallocMode
+#
+# Preallocation mode of QEMU image file
+#
+# @off: no preallocation
+# @metadata: preallocate only for metadata
+# @full: preallocate all data by calling posix_fallocate() if it is
+#        available, otherwise by writing zeros to device to ensure disk
+#        space is really available. @full preallocation also sets up
+#        metadata correctly.
+#
+# Since 2.2
+##
+{ 'enum': 'PreallocMode',
+  'data': [ 'off', 'metadata', 'full' ] }
tests/qemu-iotests/049.out +1 -1
index 71ca44d..09ca0ae 100644
@@ -179,7 +179,7 @@ qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
 Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off 
 
 qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
-qemu-img: TEST_DIR/t.qcow2: Invalid preallocation mode: '1234'
+qemu-img: TEST_DIR/t.qcow2: invalid parameter value: 1234
 Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 preallocation='1234' lazy_refcounts=off 
 
 == Check encryption option ==
Signature · 1 line
1.9.3
Reviewed-by on v13 4/6
#15 lore raw

Re: [Qemu-devel] [PATCH v13 4/6] qapi: introduce PreallocMode and a new PreallocMode full.

Quoted 58 lines · Hu Tao

This patch prepares for the subsequent patches.

---
  block/qcow2.c              | 23 +++++++++++++++--------
  qapi/block-core.json       | 16 ++++++++++++++++
  tests/qemu-iotests/049.out |  2 +-
  3 files changed, 32 insertions(+), 9 deletions(-)
block/qcow2.c +8 -8
index cf27c3f..95fb240 100644
@@ -30,6 +30,7 @@
  #include "qemu/error-report.h"
  #include "qapi/qmp/qerror.h"
  #include "qapi/qmp/qbool.h"
+#include "qapi/util.h"
  #include "trace.h"
  #include "qemu/option_int.h"
 
@@ -1738,7 +1739,7 @@ static int preallocate(BlockDriverState *bs)
 
  static int qcow2_create2(const char *filename, int64_t total_size,
                           const char *backing_file, const char *backing_format,
-                         int flags, size_t cluster_size, int prealloc,
+                         int flags, size_t cluster_size, PreallocMode prealloc,
                           QemuOpts *opts, int version,
                           Error **errp)
  {
@@ -1915,7 +1916,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
      uint64_t size = 0;
      int flags = 0;
      size_t cluster_size = DEFAULT_CLUSTER_SIZE;
-    int prealloc = 0;
+    PreallocMode prealloc = PREALLOC_MODE_OFF;
      int version = 3;
      Error *local_err = NULL;
      int ret;
@@ -1931,12 +1932,11 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
      cluster_size = qemu_opt_get_size_del(opts, BLOCK_OPT_CLUSTER_SIZE,
                                           DEFAULT_CLUSTER_SIZE);
      buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
-    if (!buf || !strcmp(buf, "off")) {
-        prealloc = 0;
-    } else if (!strcmp(buf, "metadata")) {
-        prealloc = 1;
-    } else {
-        error_setg(errp, "Invalid preallocation mode: '%s'", buf);
+    prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
+                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
          ret = -EINVAL;
          goto finish;
      }

block/qcow2.c static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) · line 1962

Quoted 5 lines
block/qcow2.c +2 -0
@@ -1958,6 +1958,13 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
          flags |= BLOCK_FLAG_LAZY_REFCOUNTS;
      }
 
+    if (prealloc && prealloc != PREALLOC_MODE_METADATA) {
+        ret = -1;

Since the return value is expected to be -errno, I'd propose "ret = -EINVAL;" here. With that fixed (or a good explanation why we want -1 here):

Quoted 45 lines
block/qcow2.c +5 -0
+        error_setg(errp, "Unsupported preallocate mode: %s",
+                   PreallocMode_lookup[prealloc]);
+        goto finish;
+    }
+
      if (backing_file && prealloc) {
          error_setg(errp, "Backing file and preallocation cannot be used at "
                     "the same time");
qapi/block-core.json +16 -0
index fb74c56..543b00b 100644
@@ -1679,3 +1679,19 @@
              'len'   : 'int',
              'offset': 'int',
              'speed' : 'int' } }
+
+# @PreallocMode
+#
+# Preallocation mode of QEMU image file
+#
+# @off: no preallocation
+# @metadata: preallocate only for metadata
+# @full: preallocate all data by calling posix_fallocate() if it is
+#        available, otherwise by writing zeros to device to ensure disk
+#        space is really available. @full preallocation also sets up
+#        metadata correctly.
+#
+# Since 2.2
+##
+{ 'enum': 'PreallocMode',
+  'data': [ 'off', 'metadata', 'full' ] }
tests/qemu-iotests/049.out +1 -1
index 71ca44d..09ca0ae 100644
@@ -179,7 +179,7 @@ qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
  Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off
 
  qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
-qemu-img: TEST_DIR/t.qcow2: Invalid preallocation mode: '1234'
+qemu-img: TEST_DIR/t.qcow2: invalid parameter value: 1234
  Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 preallocation='1234' lazy_refcounts=off
 
  == Check encryption option ==
author
#16 lore raw

Re: [Qemu-devel] [PATCH v13 4/6] qapi: introduce PreallocMode and a new PreallocMode full.

Quoted 57 lines · Max Reitz
Quoted 56 lines · Hu Tao

This patch prepares for the subsequent patches.

---
 block/qcow2.c              | 23 +++++++++++++++--------
 qapi/block-core.json       | 16 ++++++++++++++++
 tests/qemu-iotests/049.out |  2 +-
 3 files changed, 32 insertions(+), 9 deletions(-)
block/qcow2.c +8 -8
index cf27c3f..95fb240 100644
@@ -30,6 +30,7 @@
 #include "qemu/error-report.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
+#include "qapi/util.h"
 #include "trace.h"
 #include "qemu/option_int.h"
@@ -1738,7 +1739,7 @@ static int preallocate(BlockDriverState *bs)
 static int qcow2_create2(const char *filename, int64_t total_size,
                          const char *backing_file, const char *backing_format,
-                         int flags, size_t cluster_size, int prealloc,
+                         int flags, size_t cluster_size, PreallocMode prealloc,
                          QemuOpts *opts, int version,
                          Error **errp)
 {
@@ -1915,7 +1916,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     uint64_t size = 0;
     int flags = 0;
     size_t cluster_size = DEFAULT_CLUSTER_SIZE;
-    int prealloc = 0;
+    PreallocMode prealloc = PREALLOC_MODE_OFF;
     int version = 3;
     Error *local_err = NULL;
     int ret;
@@ -1931,12 +1932,11 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     cluster_size = qemu_opt_get_size_del(opts, BLOCK_OPT_CLUSTER_SIZE,
                                          DEFAULT_CLUSTER_SIZE);
     buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
-    if (!buf || !strcmp(buf, "off")) {
-        prealloc = 0;
-    } else if (!strcmp(buf, "metadata")) {
-        prealloc = 1;
-    } else {
-        error_setg(errp, "Invalid preallocation mode: '%s'", buf);
+    prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
+                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
         ret = -EINVAL;
         goto finish;
     }
@@ -1958,6 +1958,13 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
Quoted 8 lines

block/qcow2.c static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) · line 1962

Quoted 4 lines
block/qcow2.c +2 -0
         flags |= BLOCK_FLAG_LAZY_REFCOUNTS;
     }
+    if (prealloc && prealloc != PREALLOC_MODE_METADATA) {
+        ret = -1;

Since the return value is expected to be -errno, I'd propose "ret = -EINVAL;" here. With that fixed (or a good explanation why we want -1 here):

Good.

Quoted 46 lines
Quoted 43 lines
block/qcow2.c +5 -0
+        error_setg(errp, "Unsupported preallocate mode: %s",
+                   PreallocMode_lookup[prealloc]);
+        goto finish;
+    }
+
     if (backing_file && prealloc) {
         error_setg(errp, "Backing file and preallocation cannot be used at "
                    "the same time");
qapi/block-core.json +16 -0
index fb74c56..543b00b 100644
@@ -1679,3 +1679,19 @@
             'len'   : 'int',
             'offset': 'int',
             'speed' : 'int' } }
+
+# @PreallocMode
+#
+# Preallocation mode of QEMU image file
+#
+# @off: no preallocation
+# @metadata: preallocate only for metadata
+# @full: preallocate all data by calling posix_fallocate() if it is
+#        available, otherwise by writing zeros to device to ensure disk
+#        space is really available. @full preallocation also sets up
+#        metadata correctly.
+#
+# Since 2.2
+##
+{ 'enum': 'PreallocMode',
+  'data': [ 'off', 'metadata', 'full' ] }
tests/qemu-iotests/049.out +1 -1
index 71ca44d..09ca0ae 100644
@@ -179,7 +179,7 @@ qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
 Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off
 qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
-qemu-img: TEST_DIR/t.qcow2: Invalid preallocation mode: '1234'
+qemu-img: TEST_DIR/t.qcow2: invalid parameter value: 1234
 Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 preallocation='1234' lazy_refcounts=off
 == Check encryption option ==
#17 lore raw

Re: [Qemu-devel] [PATCH v13 4/6] qapi: introduce PreallocMode and a new PreallocMode full.

In reply to Hu Tao
Quoted 11 lines · Hu Tao

This patch prepares for the subsequent patches.

---
 block/qcow2.c              | 23 +++++++++++++++--------
 qapi/block-core.json       | 16 ++++++++++++++++
 tests/qemu-iotests/049.out |  2 +-
 3 files changed, 32 insertions(+), 9 deletions(-)

block/qcow2.c static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) · line 1961

Quoted 4 lines
block/qcow2.c +1 -0
@@ -1958,6 +1958,13 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         flags |= BLOCK_FLAG_LAZY_REFCOUNTS;
     }
 
+    if (prealloc && prealloc != PREALLOC_MODE_METADATA) {

I find it a bit awkward that you are checking for PREALLOC_MODE_OFF implicitly ('prealloc &&') vs. checking for prealloc mode METADATA explicitly. Since there are only three modes, would it be any simpler to just have written:

if (prealloc == PREALLOC_MODE_FULL) {

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#18 lore raw

Re: [Qemu-devel] [PATCH v13 4/6] qapi: introduce PreallocMode and a new PreallocMode full.

Quoted 11 lines · Eric Blake
Quoted 9 lines · Hu Tao

This patch prepares for the subsequent patches.

---
 block/qcow2.c              | 23 +++++++++++++++--------
 qapi/block-core.json       | 16 ++++++++++++++++
 tests/qemu-iotests/049.out |  2 +-
 3 files changed, 32 insertions(+), 9 deletions(-)
Quoted 5 lines

block/qcow2.c static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) · line 1961

Quoted 5 lines
block/qcow2.c +1 -0
@@ -1958,6 +1958,13 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         flags |= BLOCK_FLAG_LAZY_REFCOUNTS;
     }
 
+    if (prealloc && prealloc != PREALLOC_MODE_METADATA) {

This one reads as 'support PREALLOC_MODE_METADATA' only,

Quoted 7 lines

I find it a bit awkward that you are checking for PREALLOC_MODE_OFF implicitly ('prealloc &&') vs. checking for prealloc mode METADATA explicitly. Since there are only three modes, would it be any simpler to just have written:

if (prealloc == PREALLOC_MODE_FULL) {

and this one reads as 'does't support PREALLOC_MODE_FULL'. Althrough they are the same, but I'd prefer the former one. Anyway, the check is removed in patch 6.

Regards,
Hu

Quoted 5 lines
Signature · 3 lines
-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Reviewed-by on v13 4/6
#19 lore raw

Re: [Qemu-devel] [PATCH v13 4/6] qapi: introduce PreallocMode and a new PreallocMode full.

In reply to Hu Tao

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

block/qcow2.c static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) · line 1922

Quoted 13 lines

This patch prepares for the subsequent patches.

block/qcow2.c +1 -1
@@ -1915,7 +1916,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     uint64_t size = 0;
     int flags = 0;
     size_t cluster_size = DEFAULT_CLUSTER_SIZE;
-    int prealloc = 0;
+    PreallocMode prealloc = PREALLOC_MODE_OFF;
     int version = 3;
     Error *local_err = NULL;
     int ret;

The initialisation isn't really necessary any more, is it?

block/qcow2.c static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) · line 1967

Quoted 11 lines
block/qcow2.c +7 -0
@@ -1958,6 +1958,13 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         flags |= BLOCK_FLAG_LAZY_REFCOUNTS;
     }
 
+    if (prealloc && prealloc != PREALLOC_MODE_METADATA) {
+        ret = -1;
+        error_setg(errp, "Unsupported preallocate mode: %s",
+                   PreallocMode_lookup[prealloc]);
+        goto finish;
+    }
+

Like the other reviewers, I don't like this block much. But as it disappears later in the series and achieves it's job of maintaining bisectability:

author
#20 lore raw

PATCH v13 5/6 raw-posix: Add full preallocation option

In reply to Hu Tao

This patch adds a new option preallocation for raw format, and implements full preallocation.

---
 block/raw-posix.c | 92 +++++++++++++++++++++++++++++++++++++++++++------------
 qemu-doc.texi     |  8 +++++
 qemu-img.texi     |  8 +++++
 3 files changed, 88 insertions(+), 20 deletions(-)
block/raw-posix.c +72 -20
index abe0759..25f66f2 100644
@@ -30,6 +30,7 @@
 #include "block/thread-pool.h"
 #include "qemu/iov.h"
 #include "raw-aio.h"
+#include "qapi/util.h"
 
 #if defined(__APPLE__) && (__MACH__)
 #include <paths.h>
@@ -1365,6 +1366,9 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     int result = 0;
     int64_t total_size = 0;
     bool nocow = false;
+    PreallocMode prealloc = PREALLOC_MODE_OFF;
+    char *buf = NULL;
+    Error *local_err = NULL;
 
     strstart(filename, "file:", &filename);
 
@@ -1372,37 +1376,80 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
                           BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
+    buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
+    prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
+                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               &local_err);
+    g_free(buf);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        result = -EINVAL;
+        goto out;
+    }
 
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
                    0644);
     if (fd < 0) {
         result = -errno;
         error_setg_errno(errp, -result, "Could not create file");
-    } else {
-        if (nocow) {
+        goto out;
+    }
+
+    if (nocow) {
 #ifdef __linux__
-            /* Set NOCOW flag to solve performance issue on fs like btrfs.
-             * This is an optimisation. The FS_IOC_SETFLAGS ioctl return value
-             * will be ignored since any failure of this operation should not
-             * block the left work.
-             */
-            int attr;
-            if (ioctl(fd, FS_IOC_GETFLAGS, &attr) == 0) {
-                attr |= FS_NOCOW_FL;
-                ioctl(fd, FS_IOC_SETFLAGS, &attr);
-            }
-#endif
+        /* Set NOCOW flag to solve performance issue on fs like btrfs.
+         * This is an optimisation. The FS_IOC_SETFLAGS ioctl return value
+         * will be ignored since any failure of this operation should not
+         * block the left work.
+         */
+        int attr;
+        if (ioctl(fd, FS_IOC_GETFLAGS, &attr) == 0) {
+            attr |= FS_NOCOW_FL;
+            ioctl(fd, FS_IOC_SETFLAGS, &attr);
         }
+#endif
+    }
 
-        if (ftruncate(fd, total_size) != 0) {
-            result = -errno;
-            error_setg_errno(errp, -result, "Could not resize file");
-        }
-        if (qemu_close(fd) != 0) {
-            result = -errno;
-            error_setg_errno(errp, -result, "Could not close the new file");
+    if (ftruncate(fd, total_size) != 0) {
+        result = -errno;
+        error_setg_errno(errp, -result, "Could not resize file");
+        goto out_close;
+    }
+
+    if (prealloc == PREALLOC_MODE_FULL) {
+        /* posix_fallocate() doesn't set errno. */
+        result = -posix_fallocate(fd, 0, total_size);
+        if (result != 0) {
+            buf = g_malloc0(65536);
+            int64_t num = 0, left = total_size;
+
+            while (left > 0) {
+                num = MIN(left, 65536);
+                result = write(fd, buf, num);
+                if (result < 0) {
+                    result = -errno;
+                    error_setg_errno(errp, -result,
+                                     "Could not write to the new file");
+                    g_free(buf);
+                    goto out_close;
+                }
+                left -= num;
+            }
+            fsync(fd);
+            g_free(buf);
         }
+    } else if (prealloc != PREALLOC_MODE_OFF) {
+        result = -1;
+        error_setg(errp, "Unsupported preallocation mode: %s",
+                   PreallocMode_lookup[prealloc]);
+    }
+
+out_close:
+    if (qemu_close(fd) != 0 && result == 0) {
+        result = -errno;
+        error_setg_errno(errp, -result, "Could not close the new file");
     }
+out:
     return result;
 }
 
@@ -1585,6 +1632,11 @@ static QemuOptsList raw_create_opts = {
             .type = QEMU_OPT_BOOL,
             .help = "Turn off copy-on-write (valid only on btrfs)"
         },
+        {
+            .name = BLOCK_OPT_PREALLOC,
+            .type = QEMU_OPT_STRING,
+            .help = "Preallocation mode (allowed values: off, full)"
+        },
         { /* end of list */ }
     }
 };
qemu-doc.texi +8 -0
index 2b232ae..2637765 100644
@@ -527,6 +527,14 @@ Linux or NTFS on Windows), then only the written sectors will reserve
 space. Use @code{qemu-img info} to know the real size used by the
 image or @code{ls -ls} on Unix/Linux.
 
+Supported options:
+@table @code
+@item preallocation
+Preallocation mode(allowed values: @code{off}, @code{full}). An image is
+fully preallocated by calling posix_fallocate() if it's available, or by
+writing zeros to underlying storage.
+@end table
+
 @item qcow2
 QEMU image format, the most versatile format. Use it to have smaller
 images (useful if your filesystem does not supports holes, for example
qemu-img.texi +8 -0
index cb68948..063ec61 100644
@@ -418,6 +418,14 @@ Linux or NTFS on Windows), then only the written sectors will reserve
 space. Use @code{qemu-img info} to know the real size used by the
 image or @code{ls -ls} on Unix/Linux.
 
+Supported options:
+@table @code
+@item preallocation
+Preallocation mode(allowed values: @code{off}, @code{full}). An image is
+fully preallocated by calling posix_fallocate() if it's available, or by
+writing zeros to underlying storage.
+@end table
+
 @item qcow2
 QEMU image format, the most versatile format. Use it to have smaller
 images (useful if your filesystem does not supports holes, for example
Signature · 1 line
1.9.3
#21 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1422

Quoted 4 lines · Hu Tao
block/raw-posix.c +4 -0
+    if (prealloc == PREALLOC_MODE_FULL) {
+        /* posix_fallocate() doesn't set errno. */
+        result = -posix_fallocate(fd, 0, total_size);
+        if (result != 0) {

Is it better to test:

  result != ENOSYS && result != EOPNOTSUPP

here?

I think this is definitely the right approach.

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
author
#22 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Quoted 11 lines · Richard W.M. Jones

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1422

Quoted 4 lines · Hu Tao
block/raw-posix.c +4 -0
+    if (prealloc == PREALLOC_MODE_FULL) {
+        /* posix_fallocate() doesn't set errno. */
+        result = -posix_fallocate(fd, 0, total_size);
+        if (result != 0) {

Is it better to test:

  result != ENOSYS && result != EOPNOTSUPP

here?

posix_fallocate() doesn't return ENOSYS or EOPNOTSUPP. All the errors returned by posix_fallocate() apply to writing zeros, too. that is, if posix_fallocate() returns an error, we should not do writing zeros, neither.

I'm wondering what is the right way to test if posix_fallocate() is supported, something like AC_CHECK_FUNC? how?

Regards,
Hu

Quoted 12 lines

I think this is definitely the right approach.

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
author
#23 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

In reply to Richard W.M. Jones
Quoted 6 lines · Richard W.M. Jones

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1422

Quoted 4 lines · Hu Tao
block/raw-posix.c +4 -0
+    if (prealloc == PREALLOC_MODE_FULL) {
+        /* posix_fallocate() doesn't set errno. */
+        result = -posix_fallocate(fd, 0, total_size);
+        if (result != 0) {
Quoted 7 lines

Is it better to test:

  result != ENOSYS && result != EOPNOTSUPP

here?

I think this is definitely the right approach.

Hi Kevin,

How do you think this approach?

Regards,
Hu

Reviewed-by on v13 5/6
#24 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

In reply to Hu Tao
Quoted 110 lines · Hu Tao

This patch adds a new option preallocation for raw format, and implements full preallocation.

---
  block/raw-posix.c | 92 +++++++++++++++++++++++++++++++++++++++++++------------
  qemu-doc.texi     |  8 +++++
  qemu-img.texi     |  8 +++++
  3 files changed, 88 insertions(+), 20 deletions(-)
block/raw-posix.c +49 -20
index abe0759..25f66f2 100644
@@ -30,6 +30,7 @@
  #include "block/thread-pool.h"
  #include "qemu/iov.h"
  #include "raw-aio.h"
+#include "qapi/util.h"
 
  #if defined(__APPLE__) && (__MACH__)
  #include <paths.h>
@@ -1365,6 +1366,9 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
      int result = 0;
      int64_t total_size = 0;
      bool nocow = false;
+    PreallocMode prealloc = PREALLOC_MODE_OFF;
+    char *buf = NULL;
+    Error *local_err = NULL;
 
      strstart(filename, "file:", &filename);
 
@@ -1372,37 +1376,80 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
      total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
                            BDRV_SECTOR_SIZE);
      nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
+    buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
+    prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
+                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               &local_err);
+    g_free(buf);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        result = -EINVAL;
+        goto out;
+    }
 
      fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
                     0644);
      if (fd < 0) {
          result = -errno;
          error_setg_errno(errp, -result, "Could not create file");
-    } else {
-        if (nocow) {
+        goto out;
+    }
+
+    if (nocow) {
  #ifdef __linux__
-            /* Set NOCOW flag to solve performance issue on fs like btrfs.
-             * This is an optimisation. The FS_IOC_SETFLAGS ioctl return value
-             * will be ignored since any failure of this operation should not
-             * block the left work.
-             */
-            int attr;
-            if (ioctl(fd, FS_IOC_GETFLAGS, &attr) == 0) {
-                attr |= FS_NOCOW_FL;
-                ioctl(fd, FS_IOC_SETFLAGS, &attr);
-            }
-#endif
+        /* Set NOCOW flag to solve performance issue on fs like btrfs.
+         * This is an optimisation. The FS_IOC_SETFLAGS ioctl return value
+         * will be ignored since any failure of this operation should not
+         * block the left work.
+         */
+        int attr;
+        if (ioctl(fd, FS_IOC_GETFLAGS, &attr) == 0) {
+            attr |= FS_NOCOW_FL;
+            ioctl(fd, FS_IOC_SETFLAGS, &attr);
          }
+#endif
+    }
 
-        if (ftruncate(fd, total_size) != 0) {
-            result = -errno;
-            error_setg_errno(errp, -result, "Could not resize file");
-        }
-        if (qemu_close(fd) != 0) {
-            result = -errno;
-            error_setg_errno(errp, -result, "Could not close the new file");
+    if (ftruncate(fd, total_size) != 0) {
+        result = -errno;
+        error_setg_errno(errp, -result, "Could not resize file");
+        goto out_close;
+    }
+
+    if (prealloc == PREALLOC_MODE_FULL) {
+        /* posix_fallocate() doesn't set errno. */
+        result = -posix_fallocate(fd, 0, total_size);
+        if (result != 0) {
+            buf = g_malloc0(65536);
+            int64_t num = 0, left = total_size;
+
+            while (left > 0) {
+                num = MIN(left, 65536);
+                result = write(fd, buf, num);
+                if (result < 0) {
+                    result = -errno;
+                    error_setg_errno(errp, -result,
+                                     "Could not write to the new file");

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1442

Quoted 10 lines
block/raw-posix.c +9 -0
+                    g_free(buf);
+                    goto out_close;
+                }
+                left -= num;
+            }
+            fsync(fd);
+            g_free(buf);
          }
+    } else if (prealloc != PREALLOC_MODE_OFF) {
+        result = -1;

As for qcow2 in patch 4, I'd prefer -EINVAL.

Quoted 30 lines
block/raw-posix.c +14 -0
+        error_setg(errp, "Unsupported preallocation mode: %s",
+                   PreallocMode_lookup[prealloc]);
+    }
+
+out_close:
+    if (qemu_close(fd) != 0 && result == 0) {
+        result = -errno;
+        error_setg_errno(errp, -result, "Could not close the new file");
      }
+out:
      return result;
  }
 
@@ -1585,6 +1632,11 @@ static QemuOptsList raw_create_opts = {
              .type = QEMU_OPT_BOOL,
              .help = "Turn off copy-on-write (valid only on btrfs)"
          },
+        {
+            .name = BLOCK_OPT_PREALLOC,
+            .type = QEMU_OPT_STRING,
+            .help = "Preallocation mode (allowed values: off, full)"
+        },
          { /* end of list */ }
      }
  };
qemu-doc.texi +0 -0
index 2b232ae..2637765 100644

qemu-doc.texi Linux or NTFS on Windows), then only the written sectors will reserve · line 533

Quoted 7 lines
qemu-doc.texi +4 -0
@@ -527,6 +527,14 @@ Linux or NTFS on Windows), then only the written sectors will reserve
  space. Use @code{qemu-img info} to know the real size used by the
  image or @code{ls -ls} on Unix/Linux.
 
+Supported options:
+@table @code
+@item preallocation
+Preallocation mode(allowed values: @code{off}, @code{full}). An image is

Missing space in front of the opening bracket.

qemu-doc.texi Linux or NTFS on Windows), then only the written sectors will reserve · line 534

Quoted 1 line
qemu-doc.texi +1 -0
+fully preallocated by calling posix_fallocate() if it's available, or by

Hm, I personally am not so happy about contractions ("it's") in persistent documentation (even source code comments). Although I know there are already some of them in qemu-doc.texi, I'd rather avoid them. But I'll leave this up to you as I'm no native speaker.

Quoted 12 lines
qemu-doc.texi +3 -0
+writing zeros to underlying storage.
+@end table
+
  @item qcow2
  QEMU image format, the most versatile format. Use it to have smaller
  images (useful if your filesystem does not supports holes, for example
qemu-img.texi +0 -0
index cb68948..063ec61 100644
@@ -418,6 +418,14 @@ Linux or NTFS on Windows), then only the written sectors will reserve
  space. Use @code{qemu-img info} to know the real size used by the

qemu-img.texi Linux or NTFS on Windows), then only the written sectors will reserve · line 428

Quoted 10 lines
qemu-img.texi +8 -0
  image or @code{ls -ls} on Unix/Linux.
 
+Supported options:
+@table @code
+@item preallocation
+Preallocation mode(allowed values: @code{off}, @code{full}). An image is
+fully preallocated by calling posix_fallocate() if it's available, or by
+writing zeros to underlying storage.
+@end table
+

Same as for qemu-doc.texi.

However, these are all minor, so with "result = -EINVAL" and the missing space added before the brackets:

Quoted 3 lines
qemu-img.texi +0 -0
  @item qcow2
  QEMU image format, the most versatile format. Use it to have smaller
  images (useful if your filesystem does not supports holes, for example
author
#25 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Quoted 110 lines · Max Reitz
Quoted 109 lines · Hu Tao

This patch adds a new option preallocation for raw format, and implements full preallocation.

---
 block/raw-posix.c | 92 +++++++++++++++++++++++++++++++++++++++++++------------
 qemu-doc.texi     |  8 +++++
 qemu-img.texi     |  8 +++++
 3 files changed, 88 insertions(+), 20 deletions(-)
block/raw-posix.c +53 -20
index abe0759..25f66f2 100644
@@ -30,6 +30,7 @@
 #include "block/thread-pool.h"
 #include "qemu/iov.h"
 #include "raw-aio.h"
+#include "qapi/util.h"
 #if defined(__APPLE__) && (__MACH__)
 #include <paths.h>
@@ -1365,6 +1366,9 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     int result = 0;
     int64_t total_size = 0;
     bool nocow = false;
+    PreallocMode prealloc = PREALLOC_MODE_OFF;
+    char *buf = NULL;
+    Error *local_err = NULL;
     strstart(filename, "file:", &filename);
@@ -1372,37 +1376,80 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
                           BDRV_SECTOR_SIZE);
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
+    buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
+    prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
+                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               &local_err);
+    g_free(buf);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        result = -EINVAL;
+        goto out;
+    }
     fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
                    0644);
     if (fd < 0) {
         result = -errno;
         error_setg_errno(errp, -result, "Could not create file");
-    } else {
-        if (nocow) {
+        goto out;
+    }
+
+    if (nocow) {
 #ifdef __linux__
-            /* Set NOCOW flag to solve performance issue on fs like btrfs.
-             * This is an optimisation. The FS_IOC_SETFLAGS ioctl return value
-             * will be ignored since any failure of this operation should not
-             * block the left work.
-             */
-            int attr;
-            if (ioctl(fd, FS_IOC_GETFLAGS, &attr) == 0) {
-                attr |= FS_NOCOW_FL;
-                ioctl(fd, FS_IOC_SETFLAGS, &attr);
-            }
-#endif
+        /* Set NOCOW flag to solve performance issue on fs like btrfs.
+         * This is an optimisation. The FS_IOC_SETFLAGS ioctl return value
+         * will be ignored since any failure of this operation should not
+         * block the left work.
+         */
+        int attr;
+        if (ioctl(fd, FS_IOC_GETFLAGS, &attr) == 0) {
+            attr |= FS_NOCOW_FL;
+            ioctl(fd, FS_IOC_SETFLAGS, &attr);
         }
+#endif
+    }
-        if (ftruncate(fd, total_size) != 0) {
-            result = -errno;
-            error_setg_errno(errp, -result, "Could not resize file");
-        }
-        if (qemu_close(fd) != 0) {
-            result = -errno;
-            error_setg_errno(errp, -result, "Could not close the new file");
+    if (ftruncate(fd, total_size) != 0) {
+        result = -errno;
+        error_setg_errno(errp, -result, "Could not resize file");
+        goto out_close;
+    }
+
+    if (prealloc == PREALLOC_MODE_FULL) {
+        /* posix_fallocate() doesn't set errno. */
+        result = -posix_fallocate(fd, 0, total_size);
+        if (result != 0) {
+            buf = g_malloc0(65536);
+            int64_t num = 0, left = total_size;
+
+            while (left > 0) {
+                num = MIN(left, 65536);
+                result = write(fd, buf, num);
+                if (result < 0) {
+                    result = -errno;
+                    error_setg_errno(errp, -result,
+                                     "Could not write to the new file");
+                    g_free(buf);
+                    goto out_close;
+                }
+                left -= num;
Quoted 8 lines

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1442

Quoted 6 lines
block/raw-posix.c +5 -0
+            }
+            fsync(fd);
+            g_free(buf);
         }
+    } else if (prealloc != PREALLOC_MODE_OFF) {
+        result = -1;

As for qcow2 in patch 4, I'd prefer -EINVAL.

Okay.

Quoted 30 lines
Quoted 30 lines
block/raw-posix.c +14 -0
+        error_setg(errp, "Unsupported preallocation mode: %s",
+                   PreallocMode_lookup[prealloc]);
+    }
+
+out_close:
+    if (qemu_close(fd) != 0 && result == 0) {
+        result = -errno;
+        error_setg_errno(errp, -result, "Could not close the new file");
     }
+out:
     return result;
 }
@@ -1585,6 +1632,11 @@ static QemuOptsList raw_create_opts = {
             .type = QEMU_OPT_BOOL,
             .help = "Turn off copy-on-write (valid only on btrfs)"
         },
+        {
+            .name = BLOCK_OPT_PREALLOC,
+            .type = QEMU_OPT_STRING,
+            .help = "Preallocation mode (allowed values: off, full)"
+        },
         { /* end of list */ }
     }
 };
qemu-doc.texi +0 -0
index 2b232ae..2637765 100644
@@ -527,6 +527,14 @@ Linux or NTFS on Windows), then only the written sectors will reserve
Quoted 8 lines

qemu-doc.texi Linux or NTFS on Windows), then only the written sectors will reserve · line 533

Quoted 6 lines
qemu-doc.texi +4 -0
 space. Use @code{qemu-img info} to know the real size used by the
 image or @code{ls -ls} on Unix/Linux.
+Supported options:
+@table @code
+@item preallocation
+Preallocation mode(allowed values: @code{off}, @code{full}). An image is

Missing space in front of the opening bracket.

Okay. I checked that in other places of opening bracket there is a space before them.

Quoted 7 lines

qemu-doc.texi Linux or NTFS on Windows), then only the written sectors will reserve · line 533

Quoted 2 lines
qemu-doc.texi +1 -0
+fully preallocated by calling posix_fallocate() if it's available, or by

Hm, I personally am not so happy about contractions ("it's") in persistent documentation (even source code comments). Although I know there are already some of them in qemu-doc.texi, I'd rather avoid them. But I'll leave this up to you as I'm no native speaker.

I'm not, either. I don't know which one in "it's" and "it is" is more common, but I can change the contraction if it makes you feel better:)

Regards,
Hu

Quoted 33 lines
Quoted 12 lines
qemu-doc.texi +3 -0
+writing zeros to underlying storage.
+@end table
+
 @item qcow2
 QEMU image format, the most versatile format. Use it to have smaller
 images (useful if your filesystem does not supports holes, for example
qemu-img.texi +0 -0
index cb68948..063ec61 100644

qemu-img.texi Linux or NTFS on Windows), then only the written sectors will reserve · line 427

Quoted 10 lines
qemu-img.texi +8 -0
@@ -418,6 +418,14 @@ Linux or NTFS on Windows), then only the written sectors will reserve
 space. Use @code{qemu-img info} to know the real size used by the
 image or @code{ls -ls} on Unix/Linux.
+Supported options:
+@table @code
+@item preallocation
+Preallocation mode(allowed values: @code{off}, @code{full}). An image is
+fully preallocated by calling posix_fallocate() if it's available, or by
+writing zeros to underlying storage.
+@end table
+

Same as for qemu-doc.texi.

However, these are all minor, so with "result = -EINVAL" and the missing space added before the brackets:

Quoted 3 lines
qemu-img.texi +0 -0
 @item qcow2
 QEMU image format, the most versatile format. Use it to have smaller
 images (useful if your filesystem does not supports holes, for example
#26 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

In reply to Hu Tao

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

Quoted 4 lines

This patch adds a new option preallocation for raw format, and implements full preallocation.

v12 was better, it wasn't doing only metadata preallocation when you told it to do full preallocation.

Quoted 12 lines
block/raw-posix.c +12 -0
+    if (prealloc == PREALLOC_MODE_FULL) {
+        /* posix_fallocate() doesn't set errno. */
+        result = -posix_fallocate(fd, 0, total_size);
+        if (result != 0) {
+            buf = g_malloc0(65536);
+            int64_t num = 0, left = total_size;
+
+            while (left > 0) {
+                num = MIN(left, 65536);
+                result = write(fd, buf, num);
+                if (result < 0) {
+                    result = -errno;

block/raw-posix.c static int raw_create(const char *filename, QemuOpts *opts, Error **errp) · line 1440

Quoted 10 lines
block/raw-posix.c +9 -0
+                    error_setg_errno(errp, -result,
+                                     "Could not write to the new file");
+                    g_free(buf);
+                    goto out_close;
+                }
+                left -= num;
+            }
+            fsync(fd);
+            g_free(buf);
         }

This is totally pointless. If the file system doesn't support fallocate, posix_fallocate() will already try writing zeros.

Having code to write zeros in qemu only makes sense if you implement a real full preallocation mode, which doesn't use fallocate even when it's supported.

Please change the code to always write zeros for FULL, and either reintroduce FALLOC or use METADATA for the fallocate (without a fallback to zero writing code). In fact, for metadata preallocation we should probably directly use fallocate(), which has no slow zero-write fallback in the libc.

Kevin

#27 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Quoted 1 line · Kevin Wolf

Please change the code to always write zeros for FULL,

How is this useful for anyone? You don't know if the underlying SAN is going to detect these zeroes or combine these blocks together. It's just slow for no reason.

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
#28 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Am 04.09.2014 um 14:45 hat Richard W.M. Jones geschrieben:

Quoted 6 lines · Kevin Wolf
Quoted 1 line

Please change the code to always write zeros for FULL,

How is this useful for anyone? You don't know if the underlying SAN is going to detect these zeroes or combine these blocks together. It's just slow for no reason.

It's slow for the reason that the user has requested it. Do you doubt that users can know what their backend is doing? Why are you insisting on providing only the functionality that you personally need?

I doubt it's a way to make many users happier, but if you insist, we can leave full preallocation unsupported on raw-posix and allow only metadata preallocation (which should still be fallocate() without a zero-write fallback, so that you can rely on it being fast). But call it by its name and don't say "full" when you only implement "metadata".

Kevin

#29 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Quoted 11 lines · Kevin Wolf

Am 04.09.2014 um 14:45 hat Richard W.M. Jones geschrieben:

Quoted 6 lines · Kevin Wolf
Quoted 1 line

Please change the code to always write zeros for FULL,

How is this useful for anyone? You don't know if the underlying SAN is going to detect these zeroes or combine these blocks together. It's just slow for no reason.

It's slow for the reason that the user has requested it. Do you doubt that users can know what their backend is doing? Why are you insisting on providing only the functionality that you personally need?

I'm not! I'm trying to make sure we don't end up with a qemu interface which is useless for higher layers. You're proposing preallocation=full which will be slow but not actually give any guarantees, or preallocation=meta which is going to be fast but may not work, and I'm saying that's a dumb interface that's not useful.

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
#30 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Quoted 13 lines · Richard W.M. Jones
Quoted 11 lines · Kevin Wolf

Am 04.09.2014 um 14:45 hat Richard W.M. Jones geschrieben:

Quoted 6 lines · Kevin Wolf
Quoted 1 line

Please change the code to always write zeros for FULL,

How is this useful for anyone? You don't know if the underlying SAN is going to detect these zeroes or combine these blocks together. It's just slow for no reason.

It's slow for the reason that the user has requested it. Do you doubt that users can know what their backend is doing? Why are you insisting on providing only the functionality that you personally need?

Quoted 5 lines

I'm not! I'm trying to make sure we don't end up with a qemu interface which is useless for higher layers. You're proposing preallocation=full which will be slow but not actually give any guarantees, or preallocation=meta which is going to be fast but may not work, and I'm saying that's a dumb interface that's not useful.

Agree, from my POV as an app maintainer, fallocate with automatically fallback to zero'ing is really what we want to use. Having two separate options which we have to explicitly try in turn, or only having the slow option is really not what we want to use.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#31 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

In reply to Richard W.M. Jones

Am 04.09.2014 um 15:07 hat Richard W.M. Jones geschrieben:

Quoted 13 lines · Kevin Wolf
Quoted 11 lines

Am 04.09.2014 um 14:45 hat Richard W.M. Jones geschrieben:

Quoted 6 lines · Kevin Wolf
Quoted 1 line

Please change the code to always write zeros for FULL,

How is this useful for anyone? You don't know if the underlying SAN is going to detect these zeroes or combine these blocks together. It's just slow for no reason.

It's slow for the reason that the user has requested it. Do you doubt that users can know what their backend is doing? Why are you insisting on providing only the functionality that you personally need?

Quoted 5 lines

I'm not! I'm trying to make sure we don't end up with a qemu interface which is useless for higher layers. You're proposing preallocation=full which will be slow but not actually give any guarantees, or preallocation=meta which is going to be fast but may not work, and I'm saying that's a dumb interface that's not useful.

So what you propose is an interface that combines both and may unpredictably be slow or fast, and doesn't give any guarantees either. Why would this be any better?

What is your specific use case of full preallocation that wants zero writing, but only implicitly as a fallback? My expectation is that most users want cheap preallocation if it's available, but don't bother to write many gigabytes if it isn't.

Kevin

#32 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Quoted 20 lines · Kevin Wolf

Am 04.09.2014 um 15:07 hat Richard W.M. Jones geschrieben:

Quoted 13 lines · Kevin Wolf
Quoted 11 lines

Am 04.09.2014 um 14:45 hat Richard W.M. Jones geschrieben:

Quoted 6 lines · Kevin Wolf
Quoted 1 line

Please change the code to always write zeros for FULL,

How is this useful for anyone? You don't know if the underlying SAN is going to detect these zeroes or combine these blocks together. It's just slow for no reason.

It's slow for the reason that the user has requested it. Do you doubt that users can know what their backend is doing? Why are you insisting on providing only the functionality that you personally need?

Quoted 5 lines

I'm not! I'm trying to make sure we don't end up with a qemu interface which is useless for higher layers. You're proposing preallocation=full which will be slow but not actually give any guarantees, or preallocation=meta which is going to be fast but may not work, and I'm saying that's a dumb interface that's not useful.

Quoted 8 lines

So what you propose is an interface that combines both and may unpredictably be slow or fast, and doesn't give any guarantees either. Why would this be any better?

What is your specific use case of full preallocation that wants zero writing, but only implicitly as a fallback? My expectation is that most users want cheap preallocation if it's available, but don't bother to write many gigabytes if it isn't.

Stepping back, I think what we have are two general approaches:

- do the exact thing I want

- do the best effort you can

virt-manager, virt-install, virt-clone, libvirt, libguestfs, all currently do preallocation with fallback to writing data (via hand-written code). The reason is that customers using simple disks (not SANs etc) just want to be sure they're not going to run out of space during OS installation. For the vast majority of users, posix_fallocate makes this fast, but people using ext2 or *BSD get the slow-but-safe path.

Now it may be that some qemu users are going to want a very exact preallocation mode, but that doesn't mean we can't have preallocation=besteffort too.

And it's not just here. qemu has other places where we'd like "do the best thing", not "do the exact thing" ... off the top of my head:
selecting -cpu, discard support, O_DIRECT. Libguestfs has to go through hoops in these areas, often involving very hairy workarounds which reduce reliability. I'm not saying that more exact options aren't also welcome, but doing the best effort is very useful too.

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
#33 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Am 04.09.2014 um 15:43 hat Richard W.M. Jones geschrieben:

Quoted 36 lines · Kevin Wolf
Quoted 20 lines

Am 04.09.2014 um 15:07 hat Richard W.M. Jones geschrieben:

Quoted 13 lines · Kevin Wolf
Quoted 11 lines

Am 04.09.2014 um 14:45 hat Richard W.M. Jones geschrieben:

Quoted 6 lines · Kevin Wolf
Quoted 1 line

Please change the code to always write zeros for FULL,

How is this useful for anyone? You don't know if the underlying SAN is going to detect these zeroes or combine these blocks together. It's just slow for no reason.

It's slow for the reason that the user has requested it. Do you doubt that users can know what their backend is doing? Why are you insisting on providing only the functionality that you personally need?

Quoted 5 lines

I'm not! I'm trying to make sure we don't end up with a qemu interface which is useless for higher layers. You're proposing preallocation=full which will be slow but not actually give any guarantees, or preallocation=meta which is going to be fast but may not work, and I'm saying that's a dumb interface that's not useful.

Quoted 8 lines

So what you propose is an interface that combines both and may unpredictably be slow or fast, and doesn't give any guarantees either. Why would this be any better?

What is your specific use case of full preallocation that wants zero writing, but only implicitly as a fallback? My expectation is that most users want cheap preallocation if it's available, but don't bother to write many gigabytes if it isn't.

Stepping back, I think what we have are two general approaches:

- do the exact thing I want

- do the best effort you can

Quoted 8 lines

[...]

And it's not just here. qemu has other places where we'd like "do the best thing", not "do the exact thing" ... off the top of my head:
selecting -cpu, discard support, O_DIRECT. Libguestfs has to go through hoops in these areas, often involving very hairy workarounds which reduce reliability. I'm not saying that more exact options aren't also welcome, but doing the best effort is very useful too.

The point is that different people have different ideas of what "do the best thing" is. You may be surprised to learn that generally we don't roll a dice to select our defaults, but do it based on what we think is the best thing overall. And yes, because we have less information than management tools, it often means that we have to take more conservative defaults (works for everyone) than what you would want (all the features that your shiny new guest can use).

Quoted 7 lines

virt-manager, virt-install, virt-clone, libvirt, libguestfs, all currently do preallocation with fallback to writing data (via hand-written code). The reason is that customers using simple disks (not SANs etc) just want to be sure they're not going to run out of space during OS installation. For the vast majority of users, posix_fallocate makes this fast, but people using ext2 or *BSD get the slow-but-safe path.

Is this really a concern for many users? I think I never preallocated an image to make sure it fits. If anything, I run df first, but generally I assume that it works and if it doesn't, I check what happened.

Basically all users I saw asking for preallocation did it because they were worried about performance and wanted to get rid of the overhead of metadata updates during allocations.

Quoted 3 lines

Now it may be that some qemu users are going to want a very exact preallocation mode, but that doesn't mean we can't have preallocation=besteffort too.

The definition of "besteffort" depends on what you want to achieve. It is policy, and we generally try to keep policy out of qemu.

It isn't really clear that "make absolutely sure that you never get -ENOSPC on the file system level, but in the layers below it's okay, and the resulting performance doesn't matter" is the best thing for everyone.

Kevin

#34 lore raw

Re: [Qemu-devel] [PATCH v13 5/6] raw-posix: Add full preallocation option

Quoted 2 lines · Kevin Wolf

The definition of "besteffort" depends on what you want to achieve. It is policy, and we generally try to keep policy out of qemu.

I think qemu *should* have a policy of make it work and don't fail - first - and then offer a million knobs to optimize and customize things on top of that.

But whatever ... what's one more thing that we have to provide fragile workarounds for ...

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
author
#35 lore raw

PATCH v13 6/6 qcow2: Add full preallocation option

In reply to Hu Tao

preallocation=full allocates disk space by fallocating the space if posix_fallocate() is available, otherwise by writing zeros to disk to ensure disk space in any cases.

---
 block/qcow2.c              | 61 +++++++++++++++++++++++++++++++++++++++-------
 qemu-doc.texi              |  7 +++---
 qemu-img.texi              |  7 +++---
 tests/qemu-iotests/082.out | 54 ++++++++++++++++++++--------------------
 4 files changed, 87 insertions(+), 42 deletions(-)
block/qcow2.c +52 -9
index 95fb240..f222fff 100644
@@ -1772,6 +1772,56 @@ static int qcow2_create2(const char *filename, int64_t total_size,
     Error *local_err = NULL;
     int ret;
 
+    if (prealloc == PREALLOC_MODE_FULL) {
+        int64_t meta_size = 0;
+        uint64_t nreftablee, nrefblocke, nl1e, nl2e;
+        int64_t aligned_total_size = align_offset(total_size, cluster_size);
+
+        /* header: 1 cluster */
+        meta_size += cluster_size;
+
+        /* total size of L2 tables */
+        nl2e = aligned_total_size / cluster_size;
+        nl2e = align_offset(nl2e, cluster_size / sizeof(uint64_t));
+        meta_size += nl2e * sizeof(uint64_t);
+
+        /* total size of L1 tables */
+        nl1e = nl2e * sizeof(uint64_t) / cluster_size;
+        nl1e = align_offset(nl1e, cluster_size / sizeof(uint64_t));
+        meta_size += nl1e * sizeof(uint64_t);
+
+        /* total size of refcount blocks
+         *
+         * note: every host cluster is reference-counted, including metadata
+         * (even refcount blocks are recursively included).
+         * Let:
+         *   a = total_size (this is the guest disk size)
+         *   m = meta size not including refcount blocks and refcount tables
+         *   c = cluster size
+         *   y1 = number of refcount blocks entries
+         *   y2 = meta size including everything
+         * then,
+         *   y1 = (y2 + a)/c
+         *   y2 = y1 * sizeof(u16) + y1 * sizeof(u16) * sizeof(u64) / c + m
+         * we can get y1:
+         *   y1 = (a + m) / (c - sizeof(u16) - sizeof(u16) * sizeof(u64) / c)
+         */
+        nrefblocke = (aligned_total_size + meta_size + cluster_size) /
+            (cluster_size - sizeof(uint16_t) -
+             1.0 * sizeof(uint16_t) * sizeof(uint64_t) / cluster_size);
+        nrefblocke = align_offset(nrefblocke, cluster_size / sizeof(uint16_t));
+        meta_size += nrefblocke * sizeof(uint16_t);
+
+        /* total size of refcount tables */
+        nreftablee = nrefblocke * sizeof(uint16_t) / cluster_size;
+        nreftablee = align_offset(nreftablee, cluster_size / sizeof(uint64_t));
+        meta_size += nreftablee * sizeof(uint64_t);
+
+        qemu_opt_set_number(opts, BLOCK_OPT_SIZE,
+                            aligned_total_size + meta_size);
+        qemu_opt_set(opts, BLOCK_OPT_PREALLOC, PreallocMode_lookup[prealloc]);
+    }
+
     ret = bdrv_create_file(filename, opts, &local_err);
     if (ret < 0) {
         error_propagate(errp, local_err);
@@ -1877,7 +1927,7 @@ static int qcow2_create2(const char *filename, int64_t total_size,
     }
 
     /* And if we're supposed to preallocate metadata, do that now */
-    if (prealloc) {
+    if (prealloc != PREALLOC_MODE_OFF) {
         BDRVQcowState *s = bs->opaque;
         qemu_co_mutex_lock(&s->lock);
         ret = preallocate(bs);
@@ -1958,13 +2008,6 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         flags |= BLOCK_FLAG_LAZY_REFCOUNTS;
     }
 
-    if (prealloc && prealloc != PREALLOC_MODE_METADATA) {
-        ret = -1;
-        error_setg(errp, "Unsupported preallocate mode: %s",
-                   PreallocMode_lookup[prealloc]);
-        goto finish;
-    }
-
     if (backing_file && prealloc) {
         error_setg(errp, "Backing file and preallocation cannot be used at "
                    "the same time");
@@ -2525,7 +2568,7 @@ static QemuOptsList qcow2_create_opts = {
         {
             .name = BLOCK_OPT_PREALLOC,
             .type = QEMU_OPT_STRING,
-            .help = "Preallocation mode (allowed values: off, metadata)"
+            .help = "Preallocation mode (allowed values: off, metadata, full)"
         },
         {
             .name = BLOCK_OPT_LAZY_REFCOUNTS,
qemu-doc.texi +4 -3
index 2637765..b72ff7c 100644
@@ -583,9 +583,10 @@ sizes can improve the image file size whereas larger cluster sizes generally
 provide better performance.
 
 @item preallocation
-Preallocation mode (allowed values: off, metadata). An image with preallocated
-metadata is initially larger but can improve performance when the image needs
-to grow.
+Preallocation mode (allowed values: @code{off}, @code{metadata}, @code{full}).
+An image with preallocated metadata is initially larger but can improve
+performance when the image needs to grow. @code{full} preallocation is like
+the same option of @code{raw} format, but sets up metadata also.
 
 @item lazy_refcounts
 If this option is set to @code{on}, reference count updates are postponed with
qemu-img.texi +4 -3
index 063ec61..7307331 100644
@@ -474,9 +474,10 @@ sizes can improve the image file size whereas larger cluster sizes generally
 provide better performance.
 
 @item preallocation
-Preallocation mode (allowed values: off, metadata). An image with preallocated
-metadata is initially larger but can improve performance when the image needs
-to grow.
+Preallocation mode (allowed values: @code{off}, @code{metadata}, @code{full}).
+An image with preallocated metadata is initially larger but can improve
+performance when the image needs to grow. @code{full} preallocation is like
+the same option of @code{raw} format, but sets up metadata also.
 
 @item lazy_refcounts
 If this option is set to @code{on}, reference count updates are postponed with
tests/qemu-iotests/082.out +27 -27

216 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v13 6/6
#36 lore raw

Re: [Qemu-devel] [PATCH v13 6/6] qcow2: Add full preallocation option

Quoted 11 lines · Hu Tao

preallocation=full allocates disk space by fallocating the space if posix_fallocate() is available, otherwise by writing zeros to disk to ensure disk space in any cases.

---
  block/qcow2.c              | 61 +++++++++++++++++++++++++++++++++++++++-------
  qemu-doc.texi              |  7 +++---
  qemu-img.texi              |  7 +++---
  tests/qemu-iotests/082.out | 54 ++++++++++++++++++++--------------------
  4 files changed, 87 insertions(+), 42 deletions(-)
#37 lore raw

Re: [Qemu-devel] [PATCH v13 6/6] qcow2: Add full preallocation option

In reply to Hu Tao

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

Quoted 12 lines

preallocation=full allocates disk space by fallocating the space if posix_fallocate() is available, otherwise by writing zeros to disk to ensure disk space in any cases.

---
 block/qcow2.c              | 61 +++++++++++++++++++++++++++++++++++++++-------
 qemu-doc.texi              |  7 +++---
 qemu-img.texi              |  7 +++---
 tests/qemu-iotests/082.out | 54 ++++++++++++++++++++--------------------
 4 files changed, 87 insertions(+), 42 deletions(-)

block/qcow2.c static int qcow2_create2(const char *filename, int64_t total_size, · line 1823

Quoted 4 lines
block/qcow2.c +4 -0
+        qemu_opt_set_number(opts, BLOCK_OPT_SIZE,
+                            aligned_total_size + meta_size);
+        qemu_opt_set(opts, BLOCK_OPT_PREALLOC, PreallocMode_lookup[prealloc]);
+    }

This means that if used with a protocol that doesn't have a preallocation option, it gets silently ignored. I'm not completely decided yet whether that's a bug or a feature. :-)

Kevin

author
#38 lore raw

Re: [Qemu-devel] [PATCH v13 6/6] qcow2: Add full preallocation option

Quoted 13 lines · Kevin Wolf

Am 29.08.2014 um 10:33 hat Hu Tao geschrieben:

Quoted 11 lines

preallocation=full allocates disk space by fallocating the space if posix_fallocate() is available, otherwise by writing zeros to disk to ensure disk space in any cases.

---
 block/qcow2.c              | 61 +++++++++++++++++++++++++++++++++++++++-------
 qemu-doc.texi              |  7 +++---
 qemu-img.texi              |  7 +++---
 tests/qemu-iotests/082.out | 54 ++++++++++++++++++++--------------------
 4 files changed, 87 insertions(+), 42 deletions(-)
Quoted 8 lines

block/qcow2.c static int qcow2_create2(const char *filename, int64_t total_size, · line 1823

Quoted 4 lines
block/qcow2.c +4 -0
+        qemu_opt_set_number(opts, BLOCK_OPT_SIZE,
+                            aligned_total_size + meta_size);
+        qemu_opt_set(opts, BLOCK_OPT_PREALLOC, PreallocMode_lookup[prealloc]);
+    }

This means that if used with a protocol that doesn't have a preallocation option, it gets silently ignored. I'm not completely decided yet whether that's a bug or a feature. :-)

We can add code to reject preallocation option if it is determined as a bug later, but it seems not necessary currently.

Regards,
Hu