Skip to contentSkip to conversation

qemu-img: implement copy offload (-C) for dd

PATCH In review Sergio Lopez 3 participants 7 messages Feb 21, 2019 – Feb 25, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 qemu-img: implement copy offload (-C) for dd

This parameter is analogous to convert's "-C", making use of bdrv_co_copy_range().

---
 qemu-img-cmds.hx |   4 +-
 qemu-img.c       | 146 ++++++++++++++++++++++++++++++++++++-----------
 qemu-img.texi    |   2 +-
 3 files changed, 117 insertions(+), 35 deletions(-)
qemu-img-cmds.hx +2 -2
index 1526f327a5..7b85d183e5 100644
@@ -56,9 +56,9 @@ STEXI
 ETEXI
 
 DEF("dd", img_dd,
-    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [bs=block_size] [count=blocks] [skip=blocks] if=input of=output")
+    "dd [--image-opts] [-U] [-C] [-f fmt] [-O output_fmt] [bs=block_size] [count=blocks] [skip=blocks] if=input of=output")
 STEXI
-@item dd [--image-opts] [-U] [-f @var{fmt}] [-O @var{output_fmt}] [bs=@var{block_size}] [count=@var{blocks}] [skip=@var{blocks}] if=@var{input} of=@var{output}
+@item dd [--image-opts] [-U] [-C] [-f @var{fmt}] [-O @var{output_fmt}] [bs=@var{block_size}] [count=@var{blocks}] [skip=@var{blocks}] if=@var{input} of=@var{output}
 ETEXI
 
 DEF("info", img_info,
qemu-img.c +114 -32

216 lines not on this page · raw message

qemu-img.texi +1 -1
index 3b6710a580..5c5079e3bc 100644
@@ -390,7 +390,7 @@ way.
 The size can also be specified using the @var{size} option with @code{-o},
 it doesn't need to be specified separately in this case.
 
-@item dd [--image-opts] [-U] [-f @var{fmt}] [-O @var{output_fmt}] [bs=@var{block_size}] [count=@var{blocks}] [skip=@var{blocks}] if=@var{input} of=@var{output}
+@item dd [--image-opts] [-U] [-C] [-f @var{fmt}] [-O @var{output_fmt}] [bs=@var{block_size}] [count=@var{blocks}] [skip=@var{blocks}] if=@var{input} of=@var{output}
 
 Dd copies from @var{input} file to @var{output} file converting it from
 @var{fmt} format to @var{output_fmt} format.
Signature · 1 line
2.20.1
#1 lore raw
Quoted 2 lines · Sergio Lopez

This parameter is analogous to convert's "-C", making use of bdrv_co_copy_range().

The last time I tried to patch 'qemu-img dd', it was pointed out that it already has several bugs (where it is not on feature-parity with real dd), and that we REALLY want to make it a syntactic sugar wrapper around 'qemu-img convert', rather than duplicating code (which means that qemu-img convert needs to make it easier to do arbitrary offsets and subsets - although to some extent you can already do that with --image-opts and appropriate raw driver wrappers).

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html

Quoted 7 lines
---
 qemu-img-cmds.hx |   4 +-
 qemu-img.c       | 146 ++++++++++++++++++++++++++++++++++++-----------
 qemu-img.texi    |   2 +-
 3 files changed, 117 insertions(+), 35 deletions(-)

In other words, this feels like a lot of code compared to being able to just forward on to an appropriate qemu-img convert command, and I think our time would be better spent on that.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#2 lore raw
Quoted 12 lines · Eric Blake
Quoted 2 lines · Sergio Lopez

This parameter is analogous to convert's "-C", making use of bdrv_co_copy_range().

The last time I tried to patch 'qemu-img dd', it was pointed out that it already has several bugs (where it is not on feature-parity with real dd), and that we REALLY want to make it a syntactic sugar wrapper around 'qemu-img convert', rather than duplicating code (which means that qemu-img convert needs to make it easier to do arbitrary offsets and subsets - although to some extent you can already do that with --image-opts and appropriate raw driver wrappers).

Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html

Interesting, I wasn't aware of that conversation. It might a little late to go again through it, but while I don't a strong opinion about it, I do have some reservations about the idea of making 'dd' a frontend for 'convert'.

While I do see the functional similarity of both commands, to me they are quite different at a semantical level. For 'convert', I do expect it to do "the right thing" and use the optimal settings (i.e. choosing the best transfer size) by default, while 'dd' is more of "do whatever the user told you to do no matter how wrong it is".

Due to this differences, I think turning 'convert' code into something able to deal with 'dd' semantics would imply adding a considerable number of conditionals, possibly making it harder to maintain than keeping it separate.

Sergio (slp).

#3 lore raw

Am 21.02.2019 um 20:32 hat Sergio Lopez geschrieben:

Quoted 20 lines · Eric Blake
Quoted 12 lines · Sergio Lopez
Quoted 2 lines

This parameter is analogous to convert's "-C", making use of bdrv_co_copy_range().

The last time I tried to patch 'qemu-img dd', it was pointed out that it already has several bugs (where it is not on feature-parity with real dd), and that we REALLY want to make it a syntactic sugar wrapper around 'qemu-img convert', rather than duplicating code (which means that qemu-img convert needs to make it easier to do arbitrary offsets and subsets - although to some extent you can already do that with --image-opts and appropriate raw driver wrappers).

Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html

Interesting, I wasn't aware of that conversation. It might a little late to go again through it, but while I don't a strong opinion about it, I do have some reservations about the idea of making 'dd' a frontend for 'convert'.

Quoted 5 lines

While I do see the functional similarity of both commands, to me they are quite different at a semantical level. For 'convert', I do expect it to do "the right thing" and use the optimal settings (i.e. choosing the best transfer size) by default, while 'dd' is more of "do whatever the user told you to do no matter how wrong it is".

I think that's what defaults are for. It would make sense to allow the user to specify the buffer size even for 'qemu-img convert'. In fact, we already have a variable for this, we'd just have to add an option to set it explicitly instead of just relying on what the output block node tells us.

Quoted 4 lines

Due to this differences, I think turning 'convert' code into something able to deal with 'dd' semantics would imply adding a considerable number of conditionals, possibly making it harder to maintain than keeping it separate.

'qemu-img dd' currently supports five options:

* if and of. These are obviously supported for convert, too.
* count and skip. We don't have these in convert yet. We could either add the functionality there or add a raw layer in the 'dd' implementation before we call into the common code.
* bs. The buffer size is already configurable in ImgConvertState.

So getting this functionality into 'convert' should be easy.

There are more differences between 'convert' and 'dd' in how exactly the copy is done. I'm not sure whether there is actually a good use for the dumb 'dd' copying or whether it was just implemented this way because it was easier.

Currently we have features like copying only a given range only in 'dd', and most other features like zero detection, dealing with backing files, compression, copy offloading or parallel out-of-order copy only in 'convert'.

Actually, we have more than just these two places that copy images: We also have the mirror block job and for other special cases also the other block jobs that copy data, each with its own list of features and missing options.

What I really want to see eventually is a way to have all features available in all of these instead of only a random selection where you're out of luck if you want to copy part of an image with compressed output while the VM is running because these three are features supported in three different copy implementations and you can't get more than one of the features at the same time.

Kevin

#4 lore raw
Quoted 63 lines · Kevin Wolf

Am 21.02.2019 um 20:32 hat Sergio Lopez geschrieben:

Quoted 20 lines · Eric Blake
Quoted 12 lines · Sergio Lopez
Quoted 2 lines

This parameter is analogous to convert's "-C", making use of bdrv_co_copy_range().

The last time I tried to patch 'qemu-img dd', it was pointed out that it already has several bugs (where it is not on feature-parity with real dd), and that we REALLY want to make it a syntactic sugar wrapper around 'qemu-img convert', rather than duplicating code (which means that qemu-img convert needs to make it easier to do arbitrary offsets and subsets - although to some extent you can already do that with --image-opts and appropriate raw driver wrappers).

Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html

Interesting, I wasn't aware of that conversation. It might a little late to go again through it, but while I don't a strong opinion about it, I do have some reservations about the idea of making 'dd' a frontend for 'convert'.

Quoted 5 lines

While I do see the functional similarity of both commands, to me they are quite different at a semantical level. For 'convert', I do expect it to do "the right thing" and use the optimal settings (i.e. choosing the best transfer size) by default, while 'dd' is more of "do whatever the user told you to do no matter how wrong it is".

I think that's what defaults are for. It would make sense to allow the user to specify the buffer size even for 'qemu-img convert'. In fact, we already have a variable for this, we'd just have to add an option to set it explicitly instead of just relying on what the output block node tells us.

Quoted 4 lines

Due to this differences, I think turning 'convert' code into something able to deal with 'dd' semantics would imply adding a considerable number of conditionals, possibly making it harder to maintain than keeping it separate.

'qemu-img dd' currently supports five options:

* if and of. These are obviously supported for convert, too.
* count and skip. We don't have these in convert yet. We could either add the functionality there or add a raw layer in the 'dd' implementation before we call into the common code.
* bs. The buffer size is already configurable in ImgConvertState.

So getting this functionality into 'convert' should be easy.

There are more differences between 'convert' and 'dd' in how exactly the copy is done. I'm not sure whether there is actually a good use for the dumb 'dd' copying or whether it was just implemented this way because it was easier.

Currently we have features like copying only a given range only in 'dd', and most other features like zero detection, dealing with backing files, compression, copy offloading or parallel out-of-order copy only in 'convert'.

Actually, we have more than just these two places that copy images: We also have the mirror block job and for other special cases also the other block jobs that copy data, each with its own list of features and missing options.

Quoted 6 lines

What I really want to see eventually is a way to have all features available in all of these instead of only a random selection where you're out of luck if you want to copy part of an image with compressed output while the VM is running because these three are features supported in three different copy implementations and you can't get more than one of the features at the same time.

OK, makes sense. Is someone already working on this?

Sergio (slp).

#5 lore raw

Am 25.02.2019 um 14:40 hat Sergio Lopez geschrieben:

Quoted 64 lines · Kevin Wolf
Quoted 63 lines

Am 21.02.2019 um 20:32 hat Sergio Lopez geschrieben:

Quoted 20 lines · Eric Blake
Quoted 12 lines · Sergio Lopez
Quoted 2 lines

This parameter is analogous to convert's "-C", making use of bdrv_co_copy_range().

The last time I tried to patch 'qemu-img dd', it was pointed out that it already has several bugs (where it is not on feature-parity with real dd), and that we REALLY want to make it a syntactic sugar wrapper around 'qemu-img convert', rather than duplicating code (which means that qemu-img convert needs to make it easier to do arbitrary offsets and subsets - although to some extent you can already do that with --image-opts and appropriate raw driver wrappers).

Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html

Interesting, I wasn't aware of that conversation. It might a little late to go again through it, but while I don't a strong opinion about it, I do have some reservations about the idea of making 'dd' a frontend for 'convert'.

Quoted 5 lines

While I do see the functional similarity of both commands, to me they are quite different at a semantical level. For 'convert', I do expect it to do "the right thing" and use the optimal settings (i.e. choosing the best transfer size) by default, while 'dd' is more of "do whatever the user told you to do no matter how wrong it is".

I think that's what defaults are for. It would make sense to allow the user to specify the buffer size even for 'qemu-img convert'. In fact, we already have a variable for this, we'd just have to add an option to set it explicitly instead of just relying on what the output block node tells us.

Quoted 4 lines

Due to this differences, I think turning 'convert' code into something able to deal with 'dd' semantics would imply adding a considerable number of conditionals, possibly making it harder to maintain than keeping it separate.

'qemu-img dd' currently supports five options:

* if and of. These are obviously supported for convert, too.
* count and skip. We don't have these in convert yet. We could either add the functionality there or add a raw layer in the 'dd' implementation before we call into the common code.
* bs. The buffer size is already configurable in ImgConvertState.

So getting this functionality into 'convert' should be easy.

There are more differences between 'convert' and 'dd' in how exactly the copy is done. I'm not sure whether there is actually a good use for the dumb 'dd' copying or whether it was just implemented this way because it was easier.

Currently we have features like copying only a given range only in 'dd', and most other features like zero detection, dealing with backing files, compression, copy offloading or parallel out-of-order copy only in 'convert'.

Actually, we have more than just these two places that copy images: We also have the mirror block job and for other special cases also the other block jobs that copy data, each with its own list of features and missing options.

Quoted 8 lines
Quoted 6 lines

What I really want to see eventually is a way to have all features available in all of these instead of only a random selection where you're out of luck if you want to copy part of an image with compressed output while the VM is running because these three are features supported in three different copy implementations and you can't get more than one of the features at the same time.

OK, makes sense. Is someone already working on this?

I don't think anyone has found the time yet.

If you want to start work on it, maybe the best way to begin incrementally would be to factor out some common functionality from the block jobs into a new block/copy.c, from which we can later create an actual unified 'copy' block job.

Or it might turn out that another way works better. I'm only talking theory here, I haven't actually tried how well it works.

Kevin

#6 lore raw
Quoted 72 lines · Kevin Wolf

Am 25.02.2019 um 14:40 hat Sergio Lopez geschrieben:

Quoted 71 lines · Kevin Wolf
Quoted 69 lines

Am 21.02.2019 um 20:32 hat Sergio Lopez geschrieben:

Quoted 20 lines · Eric Blake
Quoted 12 lines · Sergio Lopez
Quoted 2 lines

This parameter is analogous to convert's "-C", making use of bdrv_co_copy_range().

The last time I tried to patch 'qemu-img dd', it was pointed out that it already has several bugs (where it is not on feature-parity with real dd), and that we REALLY want to make it a syntactic sugar wrapper around 'qemu-img convert', rather than duplicating code (which means that qemu-img convert needs to make it easier to do arbitrary offsets and subsets - although to some extent you can already do that with --image-opts and appropriate raw driver wrappers).

Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html

Interesting, I wasn't aware of that conversation. It might a little late to go again through it, but while I don't a strong opinion about it, I do have some reservations about the idea of making 'dd' a frontend for 'convert'.

Quoted 5 lines

While I do see the functional similarity of both commands, to me they are quite different at a semantical level. For 'convert', I do expect it to do "the right thing" and use the optimal settings (i.e. choosing the best transfer size) by default, while 'dd' is more of "do whatever the user told you to do no matter how wrong it is".

I think that's what defaults are for. It would make sense to allow the user to specify the buffer size even for 'qemu-img convert'. In fact, we already have a variable for this, we'd just have to add an option to set it explicitly instead of just relying on what the output block node tells us.

Quoted 4 lines

Due to this differences, I think turning 'convert' code into something able to deal with 'dd' semantics would imply adding a considerable number of conditionals, possibly making it harder to maintain than keeping it separate.

'qemu-img dd' currently supports five options:

* if and of. These are obviously supported for convert, too.
* count and skip. We don't have these in convert yet. We could either add the functionality there or add a raw layer in the 'dd' implementation before we call into the common code.
* bs. The buffer size is already configurable in ImgConvertState.

So getting this functionality into 'convert' should be easy.

There are more differences between 'convert' and 'dd' in how exactly the copy is done. I'm not sure whether there is actually a good use for the dumb 'dd' copying or whether it was just implemented this way because it was easier.

Currently we have features like copying only a given range only in 'dd', and most other features like zero detection, dealing with backing files, compression, copy offloading or parallel out-of-order copy only in 'convert'.

Actually, we have more than just these two places that copy images: We also have the mirror block job and for other special cases also the other block jobs that copy data, each with its own list of features and missing options.

What I really want to see eventually is a way to have all features available in all of these instead of only a random selection where you're out of luck if you want to copy part of an image with compressed output while the VM is running because these three are features supported in three different copy implementations and you can't get more than one of the features at the same time.

Quoted 8 lines
Quoted 1 line

OK, makes sense. Is someone already working on this?

I don't think anyone has found the time yet.

If you want to start work on it, maybe the best way to begin incrementally would be to factor out some common functionality from the block jobs into a new block/copy.c, from which we can later create an actual unified 'copy' block job.

Thanks for the hint. I'm going to give it a try and I'll send an early RFC patchset to put it in common and confirm we're moving in the right direction.

Sergio.