Skip to contentSkip to conversation

qemu-img: make convert async

PATCH Applied Peter Lieven 4 participants 6 messages Feb 27, 2017 – Feb 28, 2017

v2 of 4 Applied Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 qemu-img: make convert async

the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.

This can be a big performance hit when the convert process reads and writes to devices which do not benefit from kernel readahead or pagecache. In our environment we heavily have the following two use cases when using qemu-img convert.

a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backups

In both processes we use libiscsi and libnfs so we have no kernel cache.

This patch changes the convert process to work with parallel running coroutines which can significantly improve performance for network storage devices:

qemu-img (master)
nfs -> iscsi 22.8 secs
nfs -> ram 11.7 secs
ram -> iscsi 12.3 secs

qemu-img-async (8 coroutines, in-order write disabled)
nfs -> iscsi 11.0 secs
nfs -> ram 10.4 secs
ram -> iscsi 9.0 secs

This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.

---
V1->V2: - do not calculate source partition globally [Kevin]
- don't use s->status outside the global lock [Kevin]
- remove accidently left bracket in qemu-img.texi [Kevin]
- reworkd -W parageaph in documentation [Stefan]

RFC->V1: - add documentation
- add missing coroutine_fn annotation [Stefan]
- add a comment why it is safe to call coroutine_enter [Stefan]
- check -m paramater for values < 1 [Stefan]
- disallow -W parameter with compression [Stefan]

RFC V3->V4: - avoid to prepare a request queue upfront [Kevin]
- do not ignore the BLK_BACKING_FILE status [Kevin]
- redesign the interface to the read and write routines [Kevin]

RFC V2->V3: - updated stats in the commit msg from a host with a better network card
- only wake up the coroutine that is acutally waiting for a write to complete. this was not only overhead, but also breaking at least linux AIO.
- fix coding style complaints
- rename some variables and structs

RFC V1->V2: - using coroutine as worker "threads". [Max]
- keeping the request queue as otherwise it happens
that we wait on BLK_ZERO chunks while keeping the write order. it also avoids redundant calls to get_block_status and helps to skip some conditions for fully allocated imaged (!s->min_sparse)

---
 qemu-img-cmds.hx |   4 +-
 qemu-img.c       | 322 ++++++++++++++++++++++++++++++++++++++-----------------
 qemu-img.texi    |  16 ++-
 3 files changed, 243 insertions(+), 99 deletions(-)
qemu-img-cmds.hx +2 -2
index f054599..9c9702c 100644
@@ -40,9 +40,9 @@ STEXI
 ETEXI
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename")
 STEXI
-@item convert [--object @var{objectdef}] [--image-opts] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S @var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename}
+@item convert [--object @var{objectdef}] [--image-opts] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename}
 ETEXI
 
 DEF("dd", img_dd,
qemu-img.c +226 -96

481 lines not on this page · raw message

qemu-img.texi +15 -1
index 174aae3..42ea20d 100644
@@ -137,6 +137,12 @@ Parameters to convert subcommand:
 
 @item -n
 Skip the creation of the target volume
+@item -m
+Number of parallel coroutines for the convert process
+@item -W
+Allow to write out of order to the destination. This is option improves performance,
+but is only recommened for preallocated devices like host devices or other
+raw block devices.
 @end table
 
 Parameters to dd subcommand:
@@ -296,7 +302,7 @@ Error on reading data
 
 @end table
 
-@item convert [-c] [-p] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S @var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename}
+@item convert [-c] [-p] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-m @var{num_coroutines}] [-W] [-S @var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename}
 
 Convert the disk image @var{filename} or a snapshot @var{snapshot_param}(@var{snapshot_id_or_name} is deprecated)
 to disk image @var{output_filename} using format @var{output_fmt}. It can be optionally compressed (@code{-c}
@@ -326,6 +332,14 @@ skipped. This is useful for formats such as @code{rbd} if the target
 volume has already been created with site specific options that cannot
 be supplied through qemu-img.
 
+Out of order writes can be enabled with @code{-W} to improve performance.
+This is only recommended for preallocated devices like host devices or other
+raw block devices. Out of order write does not work in combination with
+creating compressed images.
+
+@var{num_coroutines} specifies how many coroutines work in parallel during
+the convert process (defaults to 8).
+
 @item dd [-f @var{fmt}] [-O @var{output_fmt}] [bs=@var{block_size}] [count=@var{blocks}] [skip=@var{blocks}] if=@var{input} of=@var{output}
 
 Dd copies from @var{input} file to @var{output} file converting it from
Signature · 1 line
1.9.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 61 lines · Peter Lieven

the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.

This can be a big performance hit when the convert process reads and writes to devices which do not benefit from kernel readahead or pagecache. In our environment we heavily have the following two use cases when using qemu-img convert.

a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backups

In both processes we use libiscsi and libnfs so we have no kernel cache.

This patch changes the convert process to work with parallel running coroutines which can significantly improve performance for network storage devices:

qemu-img (master)
nfs -> iscsi 22.8 secs
nfs -> ram 11.7 secs
ram -> iscsi 12.3 secs

qemu-img-async (8 coroutines, in-order write disabled)
nfs -> iscsi 11.0 secs
nfs -> ram 10.4 secs
ram -> iscsi 9.0 secs

This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.

---
V1->V2: - do not calculate source partition globally [Kevin]
- don't use s->status outside the global lock [Kevin]
- remove accidently left bracket in qemu-img.texi [Kevin]
- reworkd -W parageaph in documentation [Stefan]

RFC->V1: - add documentation
- add missing coroutine_fn annotation [Stefan]
- add a comment why it is safe to call coroutine_enter [Stefan]
- check -m paramater for values < 1 [Stefan]
- disallow -W parameter with compression [Stefan]

RFC V3->V4: - avoid to prepare a request queue upfront [Kevin]
- do not ignore the BLK_BACKING_FILE status [Kevin]
- redesign the interface to the read and write routines [Kevin]

RFC V2->V3: - updated stats in the commit msg from a host with a better network card
- only wake up the coroutine that is acutally waiting for a write to complete. this was not only overhead, but also breaking at least linux AIO.
- fix coding style complaints
- rename some variables and structs

RFC V1->V2: - using coroutine as worker "threads". [Max]
- keeping the request queue as otherwise it happens
that we wait on BLK_ZERO chunks while keeping the write order. it also avoids redundant calls to get_block_status and helps to skip some conditions for fully allocated imaged (!s->min_sparse)

Quoted 5 lines
---
 qemu-img-cmds.hx |   4 +-
 qemu-img.c       | 322 ++++++++++++++++++++++++++++++++++++++-----------------
 qemu-img.texi    |  16 ++-
 3 files changed, 243 insertions(+), 99 deletions(-)

I haven't checked the locking issues that Kevin pointed out, but I'm happy on the other aspects:

Applied
#2 lore raw
In reply to Peter Lieven

Am 27.02.2017 um 12:03 hat Peter Lieven geschrieben:

Quoted 27 lines

the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.

This can be a big performance hit when the convert process reads and writes to devices which do not benefit from kernel readahead or pagecache. In our environment we heavily have the following two use cases when using qemu-img convert.

a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backups

In both processes we use libiscsi and libnfs so we have no kernel cache.

This patch changes the convert process to work with parallel running coroutines which can significantly improve performance for network storage devices:

qemu-img (master)
nfs -> iscsi 22.8 secs
nfs -> ram 11.7 secs
ram -> iscsi 12.3 secs

qemu-img-async (8 coroutines, in-order write disabled)
nfs -> iscsi 11.0 secs
nfs -> ram 10.4 secs
ram -> iscsi 9.0 secs

Quoted 6 lines

This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.

Thanks, applied to the block branch.

Kevin

#3 lore raw
In reply to Peter Lieven
Quoted 8 lines · Peter Lieven

the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.

This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to

s/paremeter/parameter/

Quoted 16 lines

allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.

---

qemu-img.c +1 -1
@@ -1798,7 +1908,7 @@ static int img_convert(int argc, char **argv)
             {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
             {0, 0, 0, 0}
         };
-        c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qn",
+        c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qnm:W",
                         long_options, NULL);
         if (c == -1) {
             break;

qemu-img.c static int img_convert(int argc, char **argv) · line 2004

Quoted 5 lines
qemu-img.c +2 -0
@@ -1890,6 +2000,18 @@ static int img_convert(int argc, char **argv)
         case 'n':
             skip_create = 1;
             break;
+        case 'm':
+            num_coroutines = atoi(optarg);

atoi() should be avoided. It has no error checking, so it treats '-m 1' and '-m 1k' identically. You are a bit justified in that '-m junk' gets treated like '-m 0' and rejected, but it's still a poor error message in that case.

qemu-img.c Parameters to convert subcommand: · line 143

Quoted 13 lines
qemu-img.c +8 -0
+            if (num_coroutines < 1 || num_coroutines > MAX_COROUTINES) {
+                error_report("Allowed number of coroutines is between 1 and %d",
+                             MAX_COROUTINES);
 
+++ b/qemu-img.texi
@@ -137,6 +137,12 @@ Parameters to convert subcommand:
 
 @item -n
 Skip the creation of the target volume
+@item -m
+Number of parallel coroutines for the convert process
+@item -W
+Allow to write out of order to the destination. This is option improves performance,

Grammar suggestion:

Allow out-of-order writes to the destination. This option ...

qemu-img.texi Parameters to convert subcommand: · line 144

Quoted 1 line
qemu-img.texi +1 -0
+but is only recommened for preallocated devices like host devices or other

s/recommened/recommended/

Quoted 3 lines
qemu-img.texi +1 -0
+raw block devices.
 @end table
 
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#4 lore raw

Am 27.02.2017 um 22:09 schrieb Eric Blake:

Quoted 25 lines · Peter Lieven
Quoted 6 lines

the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.

This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to

s/paremeter/parameter/

Quoted 16 lines

allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.

---

qemu-img.c +1 -1
@@ -1798,7 +1908,7 @@ static int img_convert(int argc, char **argv)
              {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
              {0, 0, 0, 0}
          };
-        c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qn",
+        c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qnm:W",
                          long_options, NULL);
          if (c == -1) {
              break;
@@ -1890,6 +2000,18 @@ static int img_convert(int argc, char **argv)
          case 'n':
Quoted 8 lines

qemu-img.c static int img_convert(int argc, char **argv) · line 2004

Quoted 4 lines
qemu-img.c +2 -0
              skip_create = 1;
              break;
+        case 'm':
+            num_coroutines = atoi(optarg);

atoi() should be avoided. It has no error checking, so it treats '-m 1' and '-m 1k' identically. You are a bit justified in that '-m junk' gets treated like '-m 0' and rejected, but it's still a poor error message in that case.

would you use qemu_strtoul or parse_uint instead?

Kevin, shall I send a V3?

Peter

#5 lore raw

Am 28.02.2017 um 10:59 hat Peter Lieven geschrieben:

Quoted 35 lines

Am 27.02.2017 um 22:09 schrieb Eric Blake:

Quoted 25 lines · Peter Lieven
Quoted 6 lines

the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.

This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to

s/paremeter/parameter/

Quoted 16 lines

allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.

---

qemu-img.c +1 -1
@@ -1798,7 +1908,7 @@ static int img_convert(int argc, char **argv)
             {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
             {0, 0, 0, 0}
         };
-        c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qn",
+        c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qnm:W",
                         long_options, NULL);
         if (c == -1) {
             break;
@@ -1890,6 +2000,18 @@ static int img_convert(int argc, char **argv)
         case 'n':
Quoted 8 lines

qemu-img.c static int img_convert(int argc, char **argv) · line 2004

Quoted 4 lines
qemu-img.c +2 -0
             skip_create = 1;
             break;
+        case 'm':
+            num_coroutines = atoi(optarg);

atoi() should be avoided. It has no error checking, so it treats '-m 1' and '-m 1k' identically. You are a bit justified in that '-m junk' gets treated like '-m 0' and rejected, but it's still a poor error message in that case.

Quoted 3 lines

would you use qemu_strtoul or parse_uint instead?

Kevin, shall I send a V3?

If you can send a v3 today, I'll replace the patch in my queue with it. Otherwise we'll fix it on top during the freeze.

Kevin