qemu-img: make convert async
PATCH v2 1/1 qemu-img: make convert async
the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.
This can be a big performance hit when the convert process reads and writes to devices which do not benefit from kernel readahead or pagecache. In our environment we heavily have the following two use cases when using qemu-img convert.
a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backups
In both processes we use libiscsi and libnfs so we have no kernel cache.
This patch changes the convert process to work with parallel running coroutines which can significantly improve performance for network storage devices:
qemu-img (master)
nfs -> iscsi 22.8 secs
nfs -> ram 11.7 secs
ram -> iscsi 12.3 secs
qemu-img-async (8 coroutines, in-order write disabled)
nfs -> iscsi 11.0 secs
nfs -> ram 10.4 secs
ram -> iscsi 9.0 secs
This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.
---
V1->V2: - do not calculate source partition globally [Kevin]
- don't use s->status outside the global lock [Kevin]
- remove accidently left bracket in qemu-img.texi [Kevin]
- reworkd -W parageaph in documentation [Stefan]
RFC->V1: - add documentation
- add missing coroutine_fn annotation [Stefan]
- add a comment why it is safe to call coroutine_enter [Stefan]
- check -m paramater for values < 1 [Stefan]
- disallow -W parameter with compression [Stefan]
RFC V3->V4: - avoid to prepare a request queue upfront [Kevin]
- do not ignore the BLK_BACKING_FILE status [Kevin]
- redesign the interface to the read and write routines [Kevin]
RFC V2->V3: - updated stats in the commit msg from a host with a better network card
- only wake up the coroutine that is acutally waiting for a write to complete. this was not only overhead, but also breaking at least linux AIO.
- fix coding style complaints
- rename some variables and structs
RFC V1->V2: - using coroutine as worker "threads". [Max]
- keeping the request queue as otherwise it happens
that we wait on BLK_ZERO chunks while keeping the write order. it also avoids redundant calls to get_block_status and helps to skip some conditions for fully allocated imaged (!s->min_sparse)
--- qemu-img-cmds.hx | 4 +- qemu-img.c | 322 ++++++++++++++++++++++++++++++++++++++----------------- qemu-img.texi | 16 ++- 3 files changed, 243 insertions(+), 99 deletions(-)
qemu-img-cmds.hx +2 -2
@@ -40,9 +40,9 @@ STEXI ETEXI DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] filename [filename2 [...]] output_filename") + "convert [--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename") STEXI -@item convert [--object @var{objectdef}] [--image-opts] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S @var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename} +@item convert [--object @var{objectdef}] [--image-opts] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename} ETEXI DEF("dd", img_dd,
qemu-img.c +226 -96
481 lines not on this page · raw message
qemu-img.texi +15 -1
@@ -137,6 +137,12 @@ Parameters to convert subcommand: @item -n Skip the creation of the target volume +@item -m +Number of parallel coroutines for the convert process +@item -W +Allow to write out of order to the destination. This is option improves performance, +but is only recommened for preallocated devices like host devices or other +raw block devices. @end table Parameters to dd subcommand:
@@ -296,7 +302,7 @@ Error on reading data @end table -@item convert [-c] [-p] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S @var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename} +@item convert [-c] [-p] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-m @var{num_coroutines}] [-W] [-S @var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename} Convert the disk image @var{filename} or a snapshot @var{snapshot_param}(@var{snapshot_id_or_name} is deprecated) to disk image @var{output_filename} using format @var{output_fmt}. It can be optionally compressed (@code{-c}
@@ -326,6 +332,14 @@ skipped. This is useful for formats such as @code{rbd} if the target volume has already been created with site specific options that cannot be supplied through qemu-img. +Out of order writes can be enabled with @code{-W} to improve performance. +This is only recommended for preallocated devices like host devices or other +raw block devices. Out of order write does not work in combination with +creating compressed images. + +@var{num_coroutines} specifies how many coroutines work in parallel during +the convert process (defaults to 8). + @item dd [-f @var{fmt}] [-O @var{output_fmt}] [bs=@var{block_size}] [count=@var{blocks}] [skip=@var{blocks}] if=@var{input} of=@var{output} Dd copies from @var{input} file to @var{output} file converting it from
Signature · 1 line
1.9.1
Quoted 61 lines · Peter Lieven
the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.
This can be a big performance hit when the convert process reads and writes to devices which do not benefit from kernel readahead or pagecache. In our environment we heavily have the following two use cases when using qemu-img convert.
a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backupsIn both processes we use libiscsi and libnfs so we have no kernel cache.
This patch changes the convert process to work with parallel running coroutines which can significantly improve performance for network storage devices:
qemu-img (master)
nfs -> iscsi 22.8 secs
nfs -> ram 11.7 secs
ram -> iscsi 12.3 secsqemu-img-async (8 coroutines, in-order write disabled)
nfs -> iscsi 11.0 secs
nfs -> ram 10.4 secs
ram -> iscsi 9.0 secsThis patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.
Signed-off-by Peter Lieven <[email protected]>---
V1->V2: - do not calculate source partition globally [Kevin]
- don't uses->statusoutside the global lock [Kevin]
- remove accidently left bracket in qemu-img.texi [Kevin]
- reworkd -W parageaph in documentation [Stefan]
RFC->V1: - add documentation
- add missingcoroutine_fnannotation [Stefan]
- add a comment why it is safe to callcoroutine_enter[Stefan]
- check -m paramater for values < 1 [Stefan]
- disallow -W parameter with compression [Stefan]RFC
V3->V4: - avoid to prepare a request queue upfront [Kevin]
- do not ignore theBLK_BACKING_FILEstatus [Kevin]
- redesign the interface to the read and write routines [Kevin]RFC
V2->V3: - updated stats in the commit msg from a host with a better network card
- only wake up the coroutine that is acutally waiting for a write to complete. this was not only overhead, but also breaking at least linux AIO.
- fix coding style complaints
- rename some variables and structsRFC
V1->V2: - using coroutine as worker "threads". [Max]
- keeping the request queue as otherwise it happens
that we wait onBLK_ZEROchunks while keeping the write order. it also avoids redundant calls toget_block_statusand helps to skip some conditions for fully allocated imaged (!s->min_sparse)
Quoted 5 lines
--- qemu-img-cmds.hx | 4 +- qemu-img.c | 322 ++++++++++++++++++++++++++++++++++++++----------------- qemu-img.texi | 16 ++- 3 files changed, 243 insertions(+), 99 deletions(-)
I haven't checked the locking issues that Kevin pointed out, but I'm happy on the other aspects:
Kevin Wolf
AppliedAm 27.02.2017 um 12:03 hat Peter Lieven geschrieben:
Quoted 27 lines
the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.
This can be a big performance hit when the convert process reads and writes to devices which do not benefit from kernel readahead or pagecache. In our environment we heavily have the following two use cases when using qemu-img convert.
a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backupsIn both processes we use libiscsi and libnfs so we have no kernel cache.
This patch changes the convert process to work with parallel running coroutines which can significantly improve performance for network storage devices:
qemu-img (master)
nfs -> iscsi 22.8 secs
nfs -> ram 11.7 secs
ram -> iscsi 12.3 secsqemu-img-async (8 coroutines, in-order write disabled)
nfs -> iscsi 11.0 secs
nfs -> ram 10.4 secs
ram -> iscsi 9.0 secs
Quoted 6 lines
This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.
Signed-off-by Peter Lieven <[email protected]>
Thanks, applied to the block branch.
Kevin
Quoted 8 lines · Peter Lieven
the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.
This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to
s/paremeter/parameter/
Quoted 16 lines
allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.
Signed-off-by Peter Lieven <[email protected]>---
qemu-img.c +1
-1@@ -1798,7 +1908,7 @@ static int img_convert(int argc, char **argv) {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, {0, 0, 0, 0} }; - c =getopt_long (argc, argv,"hf:O:B:ce6o:s:l:S:pt:T:qn" , + c =getopt_long (argc, argv,"hf:O:B:ce6o:s:l:S:pt:T:qnm:W" , long_options,NULL );if (c ==-1 ) {break ;
qemu-img.c static int img_convert(int argc, char **argv) · line 2004
Quoted 5 lines
qemu-img.c +2
-0@@ -1890,6 +2000,18 @@ static int img_convert(int argc, char **argv)case 'n' : skip_create =1 ;break ; +case 'm' : + num_coroutines =atoi (optarg);
atoi() should be avoided. It has no error checking, so it treats '-m 1' and '-m 1k' identically. You are a bit justified in that '-m junk' gets treated like '-m 0' and rejected, but it's still a poor error message in that case.
qemu-img.c Parameters to convert subcommand: · line 143
Quoted 13 lines
qemu-img.c +8
-0+if (num_coroutines <1 || num_coroutines >MAX_COROUTINES ) { +error_report ("Allowed number of coroutines is between 1 and %d" , +MAX_COROUTINES ); +++ b/qemu-img.texi@@ -137,6 +137,12 @@ Parameters to convert subcommand: @item -n Skip the creation of the target volume +@item -m +Number of parallel coroutines for the convert process +@item -W +Allow to write out of order to the destination. This is option improves performance,
Grammar suggestion:
Allow out-of-order writes to the destination. This option ...
qemu-img.texi Parameters to convert subcommand: · line 144
Quoted 1 line
qemu-img.texi +1
-0+but is only recommened for preallocated devices like host devices or other
s/recommened/recommended/
Quoted 3 lines
qemu-img.texi +1
-0+raw block devices. @end table
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Am 27.02.2017 um 22:09 schrieb Eric Blake:
Quoted 25 lines · Peter Lieven
Quoted 6 lines
the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.
This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to
s/paremeter/parameter/
Quoted 16 lines
allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.
Signed-off-by Peter Lieven <[email protected]>---
qemu-img.c +1
-1@@ -1798,7 +1908,7 @@ static int img_convert(int argc, char **argv) {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, {0, 0, 0, 0} }; - c =getopt_long (argc, argv,"hf:O:B:ce6o:s:l:S:pt:T:qn" , + c =getopt_long (argc, argv,"hf:O:B:ce6o:s:l:S:pt:T:qnm:W" , long_options,NULL );if (c ==-1 ) {break ;@@ -1890,6 +2000,18 @@ static int img_convert(int argc, char **argv)case 'n' :
Quoted 8 lines
qemu-img.c static int img_convert(int argc, char **argv) · line 2004
Quoted 4 lines
qemu-img.c +2
-0skip_create =1 ;break ; +case 'm' : + num_coroutines =atoi (optarg);
atoi()should be avoided. It has no error checking, so it treats '-m 1' and '-m 1k' identically. You are a bit justified in that '-m junk' gets treated like '-m 0' and rejected, but it's still a poor error message in that case.
would you use qemu_strtoul or parse_uint instead?
Kevin, shall I send a V3?
Peter
Am 28.02.2017 um 10:59 hat Peter Lieven geschrieben:
Quoted 35 lines
Am 27.02.2017 um 22:09 schrieb Eric Blake:
Quoted 25 lines · Peter Lieven
Quoted 6 lines
the convert process is currently completely implemented with sync operations. That means it reads one buffer and then writes it. No parallelism and each sync request takes as long as it takes until it is completed.
This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W paremeter to
s/paremeter/parameter/
Quoted 16 lines
allow qemu-img to write to the target out of order rather than sequential. This improves performance as the writes do not have to wait for each other to complete.
Signed-off-by Peter Lieven <[email protected]>---
qemu-img.c +1
-1@@ -1798,7 +1908,7 @@ static int img_convert(int argc, char **argv) {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, {0, 0, 0, 0} }; - c =getopt_long (argc, argv,"hf:O:B:ce6o:s:l:S:pt:T:qn" , + c =getopt_long (argc, argv,"hf:O:B:ce6o:s:l:S:pt:T:qnm:W" , long_options,NULL );if (c ==-1 ) {break ;@@ -1890,6 +2000,18 @@ static int img_convert(int argc, char **argv)case 'n' :Quoted 8 lines
qemu-img.c static int img_convert(int argc, char **argv) · line 2004
Quoted 4 lines
qemu-img.c +2
-0skip_create =1 ;break ; +case 'm' : + num_coroutines =atoi (optarg);
atoi()should be avoided. It has no error checking, so it treats '-m 1' and '-m 1k' identically. You are a bit justified in that '-m junk' gets treated like '-m 0' and rejected, but it's still a poor error message in that case.
Quoted 3 lines
would you use
qemu_strtoulorparse_uintinstead?Kevin, shall I send a V3?
If you can send a v3 today, I'll replace the patch in my queue with it. Otherwise we'll fix it on top during the freeze.
Kevin